Capturing the Impacts of Archaeology for Development: Opportunities and Challenges in Evaluating the Sustainable Preservation Initiative in Peru

Sustainable Preservation Initiative Aims, Activities and Their Translation into Impacts

SPI activities are articulated as social, economic and conservation goals that mutually reinforce each other. The conservation goal, which was the organization’s starting point, is to see a reduction in the damage caused by humans within and around archaeological sites. The economic goal is to make local populations benefit from their heritage by offering employment that is directly related to the heritage value of local sites. The theory is that as local people realize that the site’s role as a job purveyor is directly linked to its touristic, cultural and historical value, they incorporate these other ways of valuing the site beyond its role as a job purveyor, and they start using the site to define their sense of identity and build a greater sense of community: this is the social goal.

To reach these objectives, SPI has assessed the needs of communities in various locations. So far, SPI’s main response to these needs has been to organize training workshops, teaching the skills needed to develop artisan businesses, including entrepreneurial skills and beyond this to develop in terms of self-esteem and confidence and broader empowerment outcomes. They work in partnership with designers to launch-site specific collections.

The main impacts identified by SPI stakeholders can be described as economic, conservation-related and social. In economic terms, the project helps artisans obtain training in the eight key skills detailed earlier and increases their visibility, which helps them find more clients. This was evidenced primarily not only in an increase in volume of sales but also in the amount of selling outlets (like shops and museums), which suggests an expanding market. Both sales and market are associated with job stability, which helps artisans plan for the future. This might be correlated to a rise in the number of artisans working with SPI, and many existing participants from Cajamarca expressed a wish to train apprentices to transmit their knowledge and expand in terms of the scale of production. A growing institutional network—for instance, links with new museums, shops or cultural institutions—was also perceived as a good proxy indicator for the expansion of both the market and the artisans’ visibility. The formalization of an association can be seen as an important step in establishing these networks and an indicator of positive impact and growing independence from SPI.

In terms of conservation, the main identified impact was a reduction in looting and encroachment activities, which is visible on the ground while assessing a site’s condition. However, the extent to which this can be attributed to SPI is hard to measure. Monitoring people’s attitudes towards their local heritage is a way to understand such a perceptual shift, which overlaps to some degree with the project’s social impacts.

In social terms, SPI fostered a sense of pride in the past and local identity among its participants. The Pachacamac entrepreneurs, women living in the vicinity of the archaeological site, who have now formed an independent cooperative named Sisan, stated that they used to see the site as a ‘pile of mud’, but they are now capable of identifying its main architectural features and explaining them to tourists. According to the María project participants, the guides at Kuelap know ‘as much as them’. The Cajamarca artisans are eager to promote their local culture, which they feel is undervalued in Peru. An association of stonemasons was started to create their own heritage-themed sculpture park, which they hoped will be a new tourist attraction in Cajamarca. SPI activities have also resulted in greater levels of social cohesion—the members of both the Pachacamac (Sisan) and María projects used common funds obtained from the project to help each other in times of need. According to an SPI staff member, the Sisan cooperative members used to experience tensions, but their level of cohesion is now greater, evidenced, for example, in their organization of polladas (food served to the neighbourhood to collect funds in times of crisis). Motivation, self-esteem and confidence have also been bolstered by the project, although the extent of this varied according to the particular community: the participants of the María project, in particular, commented on this benefit.

Beyond SPI participants, the organization’s expectation is that this system of value will resonate in the broader community, which will learn to appreciate the ways in which heritage can benefit them, and protect it, thus closing the virtuous cycle advocated by the project’s Theory of Change. These aspects are particularly difficult to measure but were visible in patterns of behaviour. In Chotuna, for instance, local people have started using the site as a background for wedding photos, which indicates greater identification with the site.

Gaps in the Theory of Change

Several gaps have been identified in the project’s Theory of Change, which may affect the extent to which intended outcomes of the projects can be realized.

The irregularity of demand from visitors due to seasonal fluctuations in tourism can lead to instability for the artisans. Many of them have identified this factor as a concern and a reason why their main commercial objective is to reach a wider market. SPI staff members recognized that artisans have lost interest in the project in the past, as economic gains may take time to fructify, and participants may become impatient. But the problem goes both ways—production can also be highly irregular, as the artisans need to adapt to the market, and to the other demands on their time and resources. As of 2018, marketing items online remained a challenge, as artisans would need to develop further skills to be able to suddenly meet heightened demand.

Another significant challenge was managing participants’ expectations. Interviews with stonemasons in Cajamarca revealed that many misunderstood the project’s objective and believed it would provide them with specific tools or allow them to go abroad for further training, both of which proved unlikely even though SPI might, in theory, at least be able to connect these groups to institutions that could support them in these ways.

The project’s aim for artisans to achieve autonomy in the long term was also affected by their initial dependence on the designs produced by associates of the organization. This was not a problem, however, as once they reach formal status as cooperatives, they can contract these designers independently of SPI as well as develop their own designs. In fact, the Sisan cooperative had already been approached by companies like the airline LATAM for new collections. Once projects had reached a ‘graduation point’—a term employed by SPI to refer to when projects reach relative independence from the organization—SPI was able to replicate its approach with other sites and communities.

The project also had unintended consequences. For instance, the María project participants faced competition from former SPI participants, other women who underwent the training workshops but decided to work on their own and selling SPI-inspired as well as cheaper items in Kuelap. SPI staff members saw these changes as a consequence of the independence they sought to establish and were working to minimize any possible negative impacts through sustained long-term communication with their participants. In most cases, they had managed to establish high levels of trust and were aware of what impacts could realistically be achieved.

Purpose of Evaluation

SPI’s evaluation strategy was aimed both at accountability and at programme understanding. The organization needed to demonstrate to its funders that its goals have been achieved, evidencing, for example, that the programme contributes to poverty reduction through increased revenue. However, SPI was also interested in attracting new donors, and as one member of staff hinted, needs to present impacts in a way that goes beyond ‘cold statistic information’ and focuses on ‘human cases’. Thus, an evaluation format that focuses on narratives, videos and photos might be easier to reach SPI’s audiences and potential donors, even though this approach is not necessarily compatible with the hard data needed.

The second objective of the evaluation strategy was to understand and improve existing projects. The 2018 monitoring strategy used sales data to understand, for instance, how much artisans produce, what products sell better in which circumstances or what the artisans need. A more inclusive and systematic evaluation strategy would help SPI understand how participants respond to the training workshops, whether the implementation needs to change and whether any negative unintended consequences can be corrected.

Sustainable Preservation Initiative’s Vision for Future Evaluation Strategies

To cater to these needs, SPI’s evaluation strategy would need to strike a balance between the organic approach favoured by some of its members, who preferred ‘informal conversations’, and a more systematic methodology. SPI also wished for a more consistent approach that they could apply to all Peruvian projects, but that allows the particularity of specific contexts to be reflected.

This could be achieved through a main, systematized framework, which can allow local differences to emerge and be taken into account centrally. Considering the size of Peru and the added strains of traveling to collect data, SPI staff members have also expressed an interest in the possibility of projects self-reporting.

Current Monitoring and Evaluation Methods

Evaluation and monitoring data were collected locally by two staff members, the project coordinator and the coordinator for commercial strategy, and transferred to the project director in the USA. SPI’s evaluation methods were skewed towards their economic objective, as their most rigorous monitoring strategy consistsed in the collection and analysis of sales data, which were collected monthly and included details on products, number of participants per site and an estimation of time spent working. In theory, this enabled the projects director to measure time-based income as a benchmark to understand how salaries have evolved over time; however, as discussed later in the article, collecting time-based data is difficult. Furthermore, using a ‘graduation point’ might provide an alternative and more objective way of measuring relative progress at a given point in time.

Conservation data were collected about once a year through interviews with site-based archaeologists and drone flights. Social data were obtained at the beginning of each project through an informal questionnaire, eliciting basic demographic data that seek to understand attitudes to local heritage. SPI’s coordinator of commercial strategy had also attempted to construct life history vignettes for the website using photography, which explored some of the social impacts of the programme. SPI’s understanding of social impacts mainly derived from site visits and informal conversations, as the initial questionnaire had not yet been repeated post-baseline.

SPI staff mentioned several ideas to improve the accuracy of data collection. The push towards more human-centred stories of impact was seen as a positive development, but one that needed systematizing in order to yield usable data. The SPI local coordinator for Cajamarca was also eager to implement short surveys for a greater understanding of social impacts. Using indicators of social and economic development was also under consideration by various staff members.

Challenges for Evaluation and Mitigation Strategies

The projects were marked by discrepancies in terms of locations and participants (gender, socio-economic status, previous experience as an artisan), affecting what activities could be conducted and at what pace and also differences in stakeholders’ expectations. For instance, most artisans sought an increase in sales and access to a greater market, but there were marked differences in group expectations. The participants of the María weaving programme expressed profound gratitude for the training workshops related to confidence raising and gender empowerment, which gave them the ‘will to work’ and encouraged them to ‘move forward as women’. The participants of the Pachacamac project were all women, most of whom had recently taken up artisan activities in a professional capacity, and they were eager to establish partnerships with national brands. The ceramicists of Cajamarca, all of whom were men who had been formalized professional artisans prior to the project, emphasized their hope to promote local culture for it to resonate across Peru and the world. Meanwhile, the stonemasons were more interested in the possibility of increasing the scale of production in order to have a reliable and stable income in order to later export internationally and to allow more apprentices to be trained. As SPI’s national director hinted, SPI was not willing to push its participants beyond what they were willing to achieve—there was no plan to turn every worker into an entrepreneur or to change their way of life. Hence, the organization saw adapting to each group’s expectations as paramount.

The perceived importance of impacts, and which ones ought to be tracked, also varied across SPI staff members. While participants in this study all recognized the social, economic and conservation relevance of the project, their priorities depended on their role within the organization, such as fundraising, managing projects on the ground or coordinating strategies at the national level. This included those who perceived archaeology as a ‘pretext’ for development, to those believing that the organization should not forget its core mission to protect heritage sites preceded any possible social impacts. This stems from a more bottom-up view of intervention, whereby SPI perceived that they should not expect participants to pursue aims differing from the ones they had prior to intervention. This view implies that the monitoring and evaluation strategy should remain simple and straightforward and focus on sales to improve existing projects. There was a degree of concern regarding the feasibility of collecting additional data and whether resources would achieve greater impact if allocated to other activities like training workshops. Other SPI stakeholders believed that the organization’s role in fostering human development and quality of life was central and even preceded site conservation. From this perspective, the social impacts observed in terms of gender empowerment, or entrepreneurial aspirations, are fundamental, and should be encouraged through workshops. This influenced the types of impacts that SPI staff wished to measure; they suggested that the evaluation strategy should be as flexible and inclusive as possible to reflect this variety of priorities.

Besides being a time-consuming activity for SPI staff, collecting data is also beset by technical difficulties. Sales data were collected at the project level, but obtaining detailed data on the time spent working has been almost impossible for cultural reasons. Artisans do not keep track of the time they spent working, which can vary depending on orders, family and social obligations. All participants were literate, but most of them had not been initially trained in formal sales accountancy although they needed these skills to formalize, pay taxes and achieve autonomy from SPI, and the learning curve could be steep. SPI staff members also worried that some artisans may be prone to representing themselves as victims in order to obtain more support, thus understating their basic earnings before joining the programme, which also made baseline data collection difficult. Others offered misinformation in the initial demographic survey, possibly through a desire to please or secure participation in the programme. For example, some stated that they were a student when they had actually been taking a short language course a few years prior, thinking they were more likely to be recruited this way. SPI staff tried to mitigate these challenges by slowly building trust with project participants. Their suggestions for future data collection plans included collecting baseline demographic data, once the design workshops had started, to allow more time for this trust to develop; designing a simpler, systematized baseline survey to collect social data; and working with local volunteers to assist permanent staff members. Creative solutions were also offered to enable time-based comparisons of economic data, despite the difficulties of obtaining a figure of hourly earnings. These included informally asking artisans to compare how much they sold at specific times of the year, like during carnival or at Christmas in order to encourage them to speak more freely about sales, or counting the number of artefacts produced in a short time frame, leading up to a fair, for example, or for a specific month.

Meanwhile, SPI’s resources to engage in additional evaluation techniques were limited as three staff members coordinated the projects and sales nationally, with the help of ‘local champions’ operating at the local level. There was no allocated budget for evaluation, since monitoring tasks were geared towards specific objectives, like understanding what products sold better under which circumstances. However, travel and monitoring funds did cover regular trips to oversee project implementation and provide evaluative feedback. These budget and personnel constraints are typical of small-scale heritage projects and should not preclude attempts to undertake rigorous evaluation. Instead, they should encourage us to think creatively about how such challenges might be navigated. Some of the methods used for evaluation in the field of international development may also shed light on possible additional solutions.

Explanatory and Ethnographic Approaches

Techniques such as realist evaluation (Pawson and Tilley, 1997; Westhorp, 2014) would potentially be a suitable fit for public archaeology projects as they strive to understand the reasons behind a project’s successes and failures by studying the specific context/mechanism/outcome configurations that enable impact. However, they require a level of time and budget commitment that is unrealistic for a small organization like SPI.

Nevertheless, detailed qualitative data are invaluable to highlight a programme’s limits and foster reflection and learning. For instance, the María project participants reported increased competition due to former project trainees who had dropped out—this significant unintended consequence might not have come to light without the focus group discussion. Currently, SPI staff access this data through informal discussions, but providing a format through which both negative and positive impacts can be monitored would ensure they are better taken into account. Ethnographic approaches have been conducted successfully in small-scale development projects with limited budgets, sometimes enlisting the help of local students (Adato, 2007). Although a fully fledged ethnographic investigation would be unrealistic due to limited staff availability, Cajamarca’s local coordinator saw the use of volunteers to implement a survey as feasible, which could potentially elicit some of the social impacts described earlier.

Indicator and Survey-Based Methods

Attempts to measure levels of poverty and well-being, through index-based surveys such as the Multidimensional Poverty Index (Alkire and Santos, 2014) and the Human Development Index (UNDP, 1990) abound in International Development. Although indicator-based approaches provide more systematized measures of economic and social development, they measure many dimensions across large populations, and data are usually collected at the national or global scale.

Smaller-scale frameworks exist, such as the Poverty Probability Index—PPI (Grameen Foundation, 2014). They are used by NGOs such as Manuela Ramos, a Peru-based women’s empowerment initiative, yet the limitations of this type of approach are exemplified by the SPI case. As discussed with staff members, SPI operated mainly in urban and peri-urban areas, where the level of poverty is comparably low. The level of education, or whether a family owns specific types of household appliances, do not seem adequate proxies for changes in socio-economic levels—most of the artisans are not in situations of extreme poverty and are likely to be above the poverty thresholds for Peru. The possibility of using PPI or other types of indicators to target future participants has been raised but is not suitable, considering that participants are selected according to their motivation and willingness to work rather than their needs (the interventions are not explicitly pro-poor), and the attrition rate is significant.

The cultural sector also endeavours to create a set of measurable indicators. The United Nations Educational, Scientific and Cultural Organization (UNESCO) has developed a set of ‘Culture for Development’ indicators broaching a variety of domains, including ‘inclusive education’, ‘gender equality outputs’ or ‘intercultural trust’ (Alonso and Medici, 2014: 13). Archaeology and heritage, however, are boiled down to a single indicator, ‘Heritage Sustainability’, which only caters to country-level initiatives, like the presence of national registries (Alonso and Medici, 2014: 131–133). Yet, despite the lack of established indicator-based frameworks, it is possible to establish a tailored survey, taking public archaeology projects’ particular impacts and stakeholders into account.

Most Significant Change Method and Life Histories

The MSC technique encourages project participants to share their experiences participating in the project, focusing on the impacts it had on their lives, through interviews (Davies and Dart, 2005). The outputs of MSC are often stories, videos or photos, and can easily be disseminated across participating communities. Additionally, and where consent has been obtained, they can be used for public relations. However, implementing the MSC technique in the field can be challenging. Indeed, in order to be rigorous, transparency is necessary, and the stories need to be verified for internal consistency and veracity (Willetts and Crawford, 2007), which involve a significant time investment.

SPI was heading into the direction of MSC through their efforts to document life histories in photographic format. Incorporating aspects of MSC might help standardize these outputs nationally. Using local production teams and building on the existing trust between local SPI representatives and project members, the creation of MSC videos would be a feasible, complementary evaluation strategy. However, this trust might reinforce the inherent bias of MSC towards positive stories.

Community-Defined Impacts: Participatory Assessment of Development and Tiny Tools

Participatory workshops elicit how impacts have been defined by project participants, and their frameworks for reporting and analysing data enable organizations to track these impacts more systematically. The ‘Participatory Assessment of Development’ (PADev) method (Dietz et al., 2013) developed around a set of exercises, elucidating what caused the changes within a community, and which ones could be traced to distinct development projects. It works well when several organizations have intervened in the same region, as is the case in many SPI locations. Each of the PADev exercises provides a framework for compilation and analysis; several of them convert qualitative data into discrete values. Visualized in tables, these changes can be documented in a more consistent manner. However, the costs of running a full PADev workshop are prohibitive: 8,000–10,000 euros are budgeted for a three-day workshop (Dietz et al., 2013: 63). To quote SPI’s national director, ‘With that money, we could run an entire project’. However, some of the exercises they offer, like the ‘Relationship between Changes and Projects’, only take about one and a half hours to run. In SPI’s case, participatory exercises could be implemented as part of late-stage SPI capacitation workshops like the ‘self-evaluation’ workshop, through which participants are encouraged to reflect on the skills they have learnt.

A similar approach is found in the ‘Tiny Tools’ participatory exercises, which are adapted from Participatory Rural Appraisal tools (Causemann et al., 2012: 3). As they operate independently, their implementation would be less cost prohibitive and time intensive, and they could be suitable for small-scale public archaeology projects. These workshops aim to understand various stakeholders’ points of view on a project’s trajectory and its possible improvements. They also produce graphs and diagrams that can be used to document a project’s outputs over time. Some of these innovative tools include a ‘Most Significant Change-light’ method, which focuses on change solely at the community level, the ‘interdependence matrix’ which produces a visual, quantified table output showing which outputs can be traced to which aspects of the project or an ‘Activity List’ that traces the importance of each activity.