Abstract
Olympic impact/legacy analysis as a research area has become a frequently discussed topic in the sociology of sport literature in general and in event management in particular since the 2000s. Although Olympic impact/legacy analysis has increased in volume, many studies still face methodological challenges (e.g. failure to adopt rigorous theoretical frameworks or overlooking additionality). The aim of this article is to unpack the logic of stakeholders in a critical evaluation of two London 2012 legacy programmes, critically reviewing the outcomes sought, and the actions adopted to achieve those outcomes, with reference to the concepts of programme theory, action theory and process tracing. We seek to redress the shortcomings of previous Olympic legacy claims which have failed to link theoretically informed accounts of the nature of Olympic-led/Olympic-inspired outcomes to explanations of the specific causal mechanisms that account for the generation of such outcomes.
Introduction
Olympic impact/legacy analysis is not new. The last decade has seen a burgeoning research interest in legacy and impact analysis for the Olympics, encompassing areas, for example, economic impacts (Blake, 2005; Kasimati and Dawson, 2009), sociocultural impacts (Truno, 1995), environmental impacts (May, 1995; Zhou et al., 2010) and political impacts (Preuss and Alfs, 2011; Tomlinson and Young, 2006).
While there has been steady growth in the amount and scope of Olympic impact/legacy analysis in the literature, it has been recognised that the operation of Olympic impact/legacy evaluation is beset by theoretical limitations and by methodological constraints. These include temporal issues concerning, for example, a lack of longitudinal studies (Karadakis and Kaplanidou, 2012) or spatial issues for research largely focusing on hosting cities or on nations as a whole (Chen, 2013). There are also challenges, for instance, due to a lack of clarity as far as defining legacy (Preuss, 2007), failure to identify the number and kinds of people affected by such legacies (Preuss, 2015), the challenge of integrating qualitative intangible evaluation results with analysis of quantitative data (VanWynsberghe, 2014) and the difficulty in estimating the extent to which emerging effects are attributable to the Games (Chen and Henry, 2016; Weed, 2012).
Parallel to the growth in studies focusing on mega-event legacies, there has been a proliferation of work promoting the application of realist-based methods of policy analysis and evaluation (Pawson and Tilley, 1997), including programme theory (Chen, 1990) and action theory (Funnell and Rogers, 2011), as well as work on process tracing (Mahoney, 2012), and on contribution analysis (Eirich et al., 2011), primarily focusing on ‘within-case’ evaluation of policy impacts. Nevertheless, despite this growth in the evaluation literature, there has been little innovation in the application of such evaluation approaches to cultural and sporting events and programmes generally, and more specifically to the Olympics Games.
In this article, we focus on two programmes which aimed to leverage benefits in a non-host region in the United Kingdom (the East Midlands) from the staging of the London 2012 Olympic and Paralympic Games in London, namely the Sport Makers and the CompeteFor programmes. In doing so, we seek to highlight the complexity in identifying the Olympics’ impacts, to undercover the logic(s) (explicit and implicit) of stakeholders (actors and institutions) in linking the outcomes sought in these two programmes to the context, and the actions adopted, to achieve these outcomes with particular reference to the application of programme theory of action and process tracing. To our knowledge, the use of a process-tracing approach (George, 1979; George and Bennett, 2005) has not yet been seriously attempted in the field of sport. The discussion of the two programmes and of the exemplary evidence collected were therefore used to demonstrate how a theory-based approach – more specifically the process-tracing approach – can help to unpack the underlying programme theory/theories and assumptions held by actors and institutions about generative causes. This approach will clarify the extent to which actions undertaken for promoting goals constitute necessary or sufficient conditions for the achievement of intended programme outcomes and, ultimately, enhance the rigour of analysis.
Olympic impact/legacy evaluations
The nature of Olympic impact/legacy analysis is considered to be a complex, wide-ranging and multi-staged process. Much of the initial work on Olympic impact analysis focused on the tangible impacts of the Games that were (apparently, at least) relatively easier to quantify, and this work mainly reflected the economic impacts of the Olympics. Commentators such as Dong and Mangan (2008), Preuss and Alfs (2011) and Waitt (2003) subsequently incorporated intangible elements of the impact/legacy into discussions and examined political, social and cultural impacts. More recently, there were two systematic reviews of major multi-sport event impacts on physical activity patterns in the general population by McCartney et al. (2010) and Weed et al. (2008). Both studies highlighted poor quality in existing studies in the field.
Methodologically speaking, there have been perhaps four principal types of approach used in the literature to investigate Olympic impact/legacy. The first involves applying economic modelling to assess economic impacts of the Olympic Games (e.g. the Olympics’ impact on gross domestic product (GDP), tourism, overall trade and national exports and employment), typically by reference to the investment in venues and in infrastructure, and mainly using the assessment techniques of the Input–Output model (Porter and Fletcher, 2008), the Cost–Benefit analysis approach (McHugh, 2006) and the Computable General Equilibrium model (Giesecke and Madden, 2007).
The evaluation difficulties associated with these approaches are derived not only from the strengths and weaknesses of the respective models, but also, in particular, from the lack of appropriate empirical justification for the use of particular multiplier estimates (Crompton, 1995), the inappropriate selection of multipliers (Porter and Fletcher, 2008) and the failure to consider opportunity costs and the substitution effects of the Games (Matheson, 2006). In addition, such economic impact studies tend to identify the relationship between, for example, inputs and outputs, without addressing in detail the identification of the generative causal explanations of how such outputs are brought about, nor the specific actions required to trigger such outputs.
The second type of legacy/impact evaluation is provided via a critical commentary on the published literature, policy documents, government reports or other management information data. This often incorporates a top-down analysis of national statistics and/or of established surveys (Carmichael et al., 2012; Veal et al., 2012). Although this type of study provides a detailed description of the policy context and allows the identification of some trends in changing legacies and impacts, the secondary data analysis is nevertheless still incapable of addressing specific policy impacts and of explaining the linked causal factors which bring about desired outcomes.
The Olympic Games Impact study (OGI study) represents a somewhat different approach. The OGI, introduced by the International Olympic Committee (IOC) in 2001 as a requirement for all host cities, forms part of the official report. It is a longitudinal analysis of change with respect to a set of 120 defined indicators of economic, sociocultural and environmental areas across a maximum period of 12 years (i.e. from 2 years prior to the Host City election until 3 years after the Olympic Games event). A clear weakness in the OGI study is that it fails to take into account the plausibility of attribution effects (VanWynsberghe, 2014). In addition, given that the OGI study (albeit a recent amendment made in 2018) is reported to the IOC on the basis of its pre-selected indicators, it is questionable in terms of its evaluation design and its lack of independence.
More recently, considerable research effort has gone into identifying legacies and impacts through case-study-based evaluation of Olympic-related programmes/interventions (Chen and Henry, 2016; Weed, 2012). Previous studies on legacy-programme evaluation have highlighted issues concerning distributional effects and attribution effects. For example, Weed (2012) used a School Sport Legacy initiative, Change 4 Life, to illustrate how to isolate the attributable effect of the London 2012 Olympic Games in the programme. This approach is allied to arguments by Habicht et al. (1999) and by VanWynsberghe (2014) about using adequacy, plausibility and probability attributions to infer the Olympics’ responsibility for specific impacts.
In summary, while the aforementioned major evaluation perspectives have, in their own ways, made many important contributions to the analysis of the Olympic impact/legacy, many existing studies only attempt to tell us whether or not impacts and legacies have been created by the Olympics, but not how those impacts were generated, by whom or/and in what circumstances. This is similar to the ‘black box’ issue which refers to the evaluation of interventions primarily focusing on assessing effects, rather than on exploring how and why those effects are produced. In other words, the precise generative mechanisms in place to produce intended outcomes remain to be elucidated. Because such causal processes have not yet been theorised explicitly, empirical analysis has been unable to unpack the mechanisms or to engage in a detailed tracing of causal processes. There is, thus, a clear need to develop a more theoretically rigorous explanatory framework to inform the evaluation of the legacy claims.
The application of theory-based evaluation in the field of sport and mega sporting events
The discussion of the theory-based, or theory-driven evaluation approach, is well documented in the literature (Chen, 1990; Donaldson, 2007). Its heuristic value has been recognised in several areas, such as health interventions (Conner et al., 1990), education programmes (Chang and Muñoz, 2006) and crime and transportation interventions (Bamberg, 2006).
However, in the field of sports studies in general and of mega sporting events more specifically, theory-based evaluation approaches have been rather limited in number and in scope. There are of course exceptions to this general pattern within the sport field, such as, Coalter’s (2007) work on programme theories in sport for development, work on the impact of physical education programmes promoting the need to adopt theories of change (Bailey et al., 2009), De Bosscher et al.’s (2008, 2015) analysis of the necessary conditions for the development of elite sport policy and Chen et al.’s (2013) analysis of meta-analysis in the evaluation of major events.
Within this handful of theory-based, sport-related, evaluation studies, two key concepts associated with theory-based approaches have been given specific attention: programme theory and causal mechanisms. Programme theory refers to a set of explicit or implicit assumptions about actions required to solve a policy problem and about why the problem will respond to such actions (Chen, 1990). Examining whether sport-for-change programmes could address racism and gang membership among at-risk youths, Coalter (2013) recognised the necessity of identifying programmes’ theoretical underpinnings as a basis for evaluation and highlighted the benefits of mapping out programme theory to assist sports programme providers in designing and delivering programmes to achieve their intended outcomes. Weed (2014) adopted programme theory to question the extent to which London 2012’s legacy strategy delivered the intended legacy outcomes; in particular, he used the theory-of-change structure to assess the efficacy and legitimacy of Olympic legacy strategy, concluding that the London 2012 legacy strategy had led to the generation of some legacy outcomes (e.g. enhanced business capacity and increased tourism spending).
Here, a distinction should be made between ‘theory of change’ and ‘theory of action’. While theory of change refers to the central mechanism by which change comes about for individuals, groups and communities, theory of action indicates how interventions are constructed to activate, or facilitate, the intended change (Funnell and Rogers, 2011). In understanding of programme theory, Funnell and Rogers (2011) suggest that, to complete the development of a programme theory, one should pull together the theory of change and the theory of action. In this article, we adopted the principles underlying theory of action to guide the process of understanding which specific approaches and actions had been taken by the stakeholders to facilitate, or bring about, the intended outcomes.
In relation to the second key concept in sports evaluation studies, namely causal mechanisms, Chen and Henry (2016) explained that understanding the processes or pathways through which an intended outcome had been generated was an important strategy for combating the aforementioned black box issue. They argued that it was not helpful only knowing whether the Olympics had increased or decreased sport participation rates nationally, without knowing the causal mechanisms for changing sport participation rates. Similarly, Hughes (2013) emphasised the importance of understanding causality in her study of the London 2012 Olympics’ sport participation legacy, advocating close examination of ‘what … processes (mechanisms) … act as the catalyst for change and in what circumstances (contexts) … the mechanisms [have an] effect (or not)’. Collectively, these studies provide important insights into how theory-based evaluations can enhance rigour in their evaluation research designs.
To critically reflect on the existing evaluation studies in the field of sport and mega sporting events, several methodological issues are worth noting. First, one of the main weaknesses of existing studies is the failure to address counterfactual scenarios (i.e. what would have happened if the intervention had not gone ahead), concepts associated with counterfactual effects and/or additionality (i.e. examination of the additional impact arising from an intervention which would not have occurred without the intervention). As Henry (2016) concluded from his involvement with the meta-evaluation study of London 2012 Olympic and Paralympic impacts, most London-2012-related programme evaluation has failed to consider the counterfactual scenario or to seriously consider the additionality impact of the Olympic Games. For example, in a commissioned evaluation for the Sport Maker programme (also cited in a subsequent section), the CFE research (Adamson and Spong, 2014) seemed to consider only the programme outcome counterfactual (that is to say, what the behaviour of Sport Makers participants would have been in the programme’s absence) and neglect the no-Games outcome counterfactual (that is to say, what Sport Makers participants’ behaviour would have been if the London 2012 Games had not happened).
The second methodological issue worth noting, in critically reflecting on the relevant sports evaluation studies, is that there is still a lack of recognition of the contextual influences for a sports programme/intervention. Actions within a programme can cause success or failure to achieve outcomes, but success in invoking a causal mechanism will vary from one context to another. Evaluations of sports programmes/interventions, particularly the evaluations employing experimental evaluation design (Fuller et al., 2010; Lubans and Morgan, 2008), tend to ignore the influence of contexts (social and cultural conditions in which programmes are implemented) on the impacts/outcomes of a programme. This lack of attention to the social and cultural conditions – which precede programmes and continue during their implementation – can be potentially problematic (Pawson and Tilley, 1997), as we do not know the conditions necessary for change mechanisms to operate.
The third issue is that most studies in the field of sport and mega sporting events tend to be outcome-based evaluations (name deleted to maintain the integrity of the review process, Chen, 2018), paying little attention to ‘which factors of a sport programme/intervention have caused particular outcomes in given contexts’ (Chen, 2018: 10). Although theory-based evaluation approaches have gradually gained popularity in the field of sport, their application is still limited both in number and scope and not yet well established.
Process tracing
In the context of sport, evaluation research is still in its infancy (Chen, 2018). Few studies, if any, adopt a process-tracing approach. This article thus aims to use two Olympic programmes to illustrate the feasibility of using process tracing, how this would have been done and how it would have enhanced evaluations.
The process-tracing approach (George, 1979; George and Bennett, 2005) is viewed as particularly useful for measuring and testing hypothesised causal mechanisms (Beach and Pedersen, 2013; Bennett and Checkel, 2014), by tracing whether ‘there is actual within-case process-related evidence of a theorized mechanism actually operating as predicted in the chosen case’ (Beach, 2018: 66). The general consensus in the literature is that the core of process-tracing focuses on tracing causal mechanisms (Bennett and Checkel, 2014; George and Bennett, 2005; Waldner, 2012). Such an approach builds on a mechanism-based understanding of causation, in contrast to a counterfactual understanding of causation which is often employed in comparative case study methods (Beach and Pedersen, 2016).
In the context of process tracing, Bayesian probability is often used as an inferential logic that underlies deductive process tracing (Bennett and Checkel, 2014). Bayesian logic deals with probability inference. Specifically, in order to evaluate the probability of a hypotheses, Bayesian logic looks to specify some prior probability, which is then updated to a posteriori probability when new data/evidence is presented (The notes related to Bayes theorem were edited and published 2 years after Bayes’ death in 1761 by Richard Price, see Bayes and Price, 1763). As suggested by Bennett and Checkel (2014), using Bayesian logic explicitly in process tracing is valuable, as it asks researchers to make specific and transparent assumptions – regarding the probability of finding a certain kind of evidence (being conditional on both the truth and the falsity of a theory) – so that such assumptions-specifying-processes can push researchers to clarify their own thinking and to be more consistent in their logic as they update their confidence in alternative theories and explanations (Bennett, 2014).
The application of process tracing begins with the establishment of a theoretical hypothesis. It then assesses how confident researchers can be in the validity of each causal hypothesis based on existing research, or even on common sense understandings. In other words, what existing research tells us about the plausibility of the causal relationship is being presented in the case in question (i.e. prior confidence in the theoretical H). The level of prior confidence influences how the impact of the evidence should be evaluated – whether we should focus on collecting confirming or disconfirming evidence (Beach and Pedersen, 2016): if the prior confidence is high, it would be more useful to collect disconfirming evidence in an attempt to potentially learn something new; whereas, if the prior evidence is low, any confirming evidence (even relatively weak) would increase our confidence. Here, the confirming power of evidence relates to theoretical uniqueness (i.e. the expected probability of finding that predicted evidence if the H is not true); the disconfirming power of evidence is associated with theoretical certainty (i.e. the probability of observing the evidence conditional upon the hypothesis being true). Thus, based on the four possible combinations of (non-)uniqueness and (un)certainty, Van Evera (1997) developed four useful probative value tests in process tracing: (1) Hoop tests consider evidence that is certain but not unique, which is the most useful test in excluding alternative hypotheses and (2) smoking-gun tests involve evidence that is unique but not certain. As explained by Van Evera, the fact that a suspect holds a smoking gun in his or her hands after a murder can strongly affirm the suspect; however, the absence of a smoking gun does not exonerate the suspect because that the murderer could simply hide the gun. (3) Doubly decisive tests are both unique and certain; in other words, the evidence used is necessary and sufficient to provide great confidence in an explanation. (4) Straw-in-the-wind tests use evidence that is neither unique nor certain, which provides the weakest evidence of the four tests.
As argued above, when assessing mega sporting events (e.g. the Olympics), the use of a case study approach is often criticised as being ambiguous in providing causal contribution claims. In this respect, the process-tracing type of causal case study approaches, which engage a theory-based evaluation framework (either theory-testing or theory-building), can be useful, as indeed can ‘explaining-outcome process-tracing’. Beach and Pedersen (2013) describe this third process-training approach, explaining-outcome process-tracing attempts to craft a minimally sufficient explanation of a puzzling outcome in a specific historical case. Here the aim is not to build or test more general theories but to craft a (minimally) sufficient explanation of the outcome of the case where the ambitions are more case-centric than theory-oriented. (p. 3)
In a recent evaluation of a mega cultural event, that is, the 2015 Universal Exposition, Busetti and Dente (2017) used process tracing to search, collect and assess project evidence in order to identify the possible causal impacts of the event in relation to the two projects. Their evaluation demonstrated the feasibility of applying process tracing to mega events evaluation, noted the improvement of rigour and transparency in within-case inferences and particularly concluded Bayesian theory’s capability of increasing reliability when assessing less straightforward causal attributions.
The two Olympic legacy-programme evaluations
Although there are widespread claims in terms of legacy benefits derived from the staging of the Olympic Games, there has been little attempt to identify in detail the explanations of generative causes of outcomes. For example, claims that the inspirational effect of staging and witnessing the Games will result in increased participation in sport among the wider population have generally failed to articulate and test causal explanations of the process of achieving desired outcomes. Similarly, claims about the economic impacts (positive or negative) of the Games have tended to rely on statistical association of input and output measures rather than on identifying and testing causal accounts. Thus, the two cases we have selected for evaluation relate to using the occasion of staging the 2012 Olympics to generate positive outcomes in relation to sport volunteering, and to generating competitiveness in regional organisations bidding for London 2012 contracts. The first of these is the Sport Makers programme designed to increase sport volunteering and thereby to enhance the supply of sporting opportunities, and ultimately the level of sport participation in the general population. The second concerns the CompeteFor programme designed to enhance the competitive position of regional supplier organisations in competing for Olympic contracts, and thus to have positive economic impacts.
The methodology adopted in the investigation of each of these case study programmes involved an initial review of policy and promotional literature, surveys for Sport Makers programme and interviews with key stakeholders for both programmes.
The subsequent analysis of data consisted of three stages. The first of these was to construct a logic model for each of the programmes indicating context, inputs, throughputs, outputs and outcomes and the causal linkages between each of these stages. This in effect constituted the programme theories (Donaldson, 2007) underpinning the respective programmes derived from analysis of the policy documents and stakeholder interviews. The second stage involved identifying the action theory (Funnell and Rogers, 2011) providing the rationale for the actions taken by the stakeholders to initiate the causal linkages between each of the stages. The third stage involved employing a process-tracing approach (Mahoney, 2012) to identify the extent to which actions undertaken constituted necessary or sufficient conditions of the achievement of the outcomes sought.
The Sport Makers case
The Sport Makers programme was one of the major national volunteering programmes related to the London 2012 Olympic Games. This volunteering programme was designed to harness the inspiration of the London 2012 Olympic Games to recruit, train and deploy volunteers, and it ran from July 2011 to September 2013.
Each of the Sport Makers was given an introduction to the programme and to sports volunteering via a series of workshops delivered locally by a training provider and in conjunction with a County Sport Partnership. In particular, an understanding of the nature and significance and implementation of Olympic and Paralympic values were key aspects of volunteer training delivered via onsite workshops. Sport Makers, once trained, were then supported in finding local opportunities to facilitate sport among their friends and in their communities generally. Thus, its broad strategic aim was to increase the number of people volunteering in sport and thus to facilitate sport participation and the quality of sporting opportunities for the purpose of creating a tangible sporting and volunteering legacy of the London 2012 Games.
Three methods were used for collecting data: participant questionnaire surveys (with 548 people taking part in the Sport Maker programme in Leicestershire, 17% completed the survey; n = 94, thus with a 95% confidence level, the confidence interval is a maximum ± 9.2%), interviews with key delivery partners (including a series of consultations with the sub-regional Sport Makers programme leader from January to November 2012 and one in-depth interview with a sub-regional key stakeholder who supported promotion of the programme) and in a review of internal policy and external marketing materials. Specifically, the analysis of key strategic and programme-related documents was conducted to gain an overall understanding of the context of the evaluation, to support the design of the questionnaires and interviewer questions and for situation analysis; surveys were used to collect empirical data from programme participants about their motives for engaging with the programme and for being involved with the Games; a series of stakeholder interviews and consultations were carried out to verify and respond to issues emerging from document analysis and more importantly to analyse explicitly the nature and coherence of the approach adopted to achieve the goals of the programme, the underlying programme theory and stakeholders’ assumptions about generative causes.
Figure 1 outlines a logic model for the Sport Makers programme, which summarises the logic of actions and outcomes, and stakeholders’ assumptions about the theory of change underpinning the programme. Regarding the first causal assumption of the programme, it was anticipated that, people, who had not had any previous volunteering experiences and might not yet have had the confidence and required skills for volunteering (i.e. new volunteers), would be inspired by the London 2012 Games and motivated to be part of the Games and the sports system through volunteering, and thus the offering of training sessions, within which the Olympic and Paralympic values and the value of sport were embedded, would inspire and support those new volunteers in terms of skill development and subsequently would increase the likelihood of them stepping into voluntary work. The actions required for achieving this included the promoting of the Sport Makers programme, organising training sessions and developing an appropriate educational package.
Programme and action logic model for the Leicestershire Sport Makers programme.
The second element of programme theory was the assumption that this body of trained Sport Makers would feel confident in participating in local volunteering activities, and would subsequently lead and organise their own volunteer work (e.g. organising sport for their friends and helping local sports clubs or events), and thus the number of hours of volunteering activities in the local sports system would be increased. At the level of action theory, ‘menus’ of volunteering opportunities to programme participants (the menu consisting of types of organisation and types of volunteering activity for which there was an identified demand) and a VolunteerWeb platform for recording Sport Makers’ voluntary hours would be required.
The third element in programme theory was the assumption that, through this continued involvement with the programme and community volunteering, this group of Sport Makers would be maintained as a long-term local volunteer resource and would ensure sustainable volunteering legacies at a community level. Here, the Sport Makers website and the VolunteerWeb platform were resources to facilitate long-term voluntary commitment and engagement.
Moving on from programme theory and action theory, we proceed to process tracing, which employs generative causal frameworks to evaluate the strength of causal claims. The central hypothesis we wish to test is as follows:
H1: The staging of the Olympic Games has motivated new volunteers to seek out or to take advantage of sports volunteer training and volunteering opportunities promoted and facilitated by a volunteering programme (i.e. the Sport Makers programme).
The document analysis and a preliminary interview with the Leicestershire Sport Makers’ programme leader determined prior confidence in H1 as moderate: (1) The Sport Makers programme marketing materials and the detailed programme description explicitly acknowledged the assumption that the Olympics would catalyse interest in volunteering, the programme intertwined with Olympics-related messages and offered inspiration training workshops to ready new volunteers for volunteering actions. Hence, document analysis demonstrated high prior confidence in H1. (2) However, the programme leader did not consider the programme to be different from previous programmes. For her, it was more relevant that previous volunteering programmes had stopped running and the team now focused on promoting Sport Makers programme. Thus, the preliminary interview with the programme stakeholder demonstrated low prior confidence in H1.
Table 1 summarises the relevant evidence needed to increase confidence in H1, the certainty and uniqueness value possessed by each piece of evidence, whether the test was passed or failed and how the individual test affected confidence in H1. As can be seen from Table 1, although H1 passed the first three tests – the Sport Makers programme was a London 2012 legacy programme (E1.2), it was launched during the London 2012 Games period (E1.1) and volunteers did sign up for the programme (E1.3) – these tests exhibited low uniqueness and low certainty for confirming H1. Although the Olympic Games took place but cannot be viewed as a necessary or sufficient condition for motivation to join volunteer training, the three tests provided some weak evidence to support the hypothesis. The results of our Sport Makers participant survey also showed that a large proportion of those enrolling were not new volunteers (58 out of 94 survey respondents were not new volunteers); there was no evidence of sustained increases or positive spikes in the number of people registered to the programme prior to, during and after the London 2012 Games (E1.4).
Assessing the evidence for H1 (the Sport Makers programme).
The two smoking-gun tests – confirming the inspirational effect of the Olympics on volunteers’ programme participation (E1.5 and E1.6) – exhibited some uniqueness: Although the significance of the Games was not the most important motivational factor (ranked 4th out of 11 motivational factors) in the decision to join Sport Makers, it was nevertheless an important factor (with a mean of 4.25 on a 5-point Likert-type scale) and, in particular, it ranked slightly more highly for importance than other factors did for people who were new volunteers. The difference in means between those with and those without volunteer experience was statistically insignificant.
In relation to the longer-term outcomes sought, we did not have evidence of the extent to which people with no previous volunteering experience were inspired by the Olympics to engage with the programme or if people continued to participate in volunteering work after the Olympics (E1.7); nor do we know whether training provided by the programme was sufficient or relevant enough for recipients to operate effectively in the field of sport volunteering (E1.8). Two other studies, published after our own evaluation was completed, are relevant in terms of addressing these issues. Nicholls et al. (2013) conclude on the basis of qualitative case studies of the Sport Makers programme that matching young people’s needs as new volunteers with the experience offered by the sports organisations in which they were placed, and the training they received, was not generally effective. A second study, the CFE evaluation of the programme nationally (Adamson and Spong, 2014), which did address the issue of sustainability of the scheme in terms of outcomes, made optimistic claims about short-term sustainability: The sustainability of participation in Sport Maker was high with 59.2% of Sport Makers stating that at least three-quarters of the people they recruited were still participating in sport and physical activities after seven months.
However, the problem here is that we still do not know how many of the 59.2 per cent of Sport Makers were inspired by the 2012 Games or sought sustained involvement in volunteering as a result. Furthermore, such self-reporting of intentions is not always a reliable indicator of longer-term behavioural outcomes and in addition, the Nicholls et al.’s (2013) study reports highly unreliable recording of data concerning the level and length of volunteer activity.
Thus, we can conclude from the programme, action and process tracing elements that there is weak positive support for claims about the impact of staging the Games on motivation to engage with volunteering and the evidence in relation to the effectiveness and sustainability of activities is such that support for the achievement of longer-term outcomes is not apparent.
The CompeteFor case
CompeteFor was designed as a free service with three interrelated elements. The first was an electronic brokerage system to link those organisations seeking to contract to buy goods and services (buyer organisations) relating to the staging of the London 2012 Games and other public service contracts from supplier organisations. In the supply chain for the provision of such goods and services, there were three (or more) tiers of organisations: those directly contracted by London 2012 (or other public-sector entities) to provide goods and services (Tier 1 organisations), those organisations given a sub-contract to serve the needs of the Tier 1 bodies (i.e. Tier 2 organisations) and those subcontracted to supply goods and services to Tier 2 organisations (i.e. Tier 3 organisations).
The London 2012 Olympics is here presented as an incentive opportunity for encouraging businesses to engage with the CompeteFor programme. CompeteFor acted as a brokerage between buyer and supplier organisations in a number of ways. First, it established a web portal on which it sought to register information on all London-2012-related contracts (Tiers 1, 2 and 3) for which supplier organisations might compete. Second, it sought to encourage potential supplier organisations to register their interest in bidding for such contracts and subsequently to publish their profile on the portal. In order to publish its profile, an organisation had to provide core data about itself (e.g. the size of workforce, quality of Information Communication Technology, extent of insurance cover etc.) to provide a clear indication of organisational professionalism and capacity. Third, CompeteFor fostered the publication by buyer organisations of calls for interest in relation to contracts for the purchase of products and/or services. Fourth, CompeteFor also linked businesses to existing business support services. Non-UK companies can access the CompeteFor website, but are not given support services.
Supplier engagement activities sought to maximise the number of supplier and potential supplier organisations registering and publishing profiles on the portal and to promote awareness of Olympic-related contract opportunities such that suppliers would have an increased chance of being shortlisted for 2012. Facilitated by regional Business Links, each region was also charged with responsibility of providing and funding regional level supplier awareness and skill development, with delivery of training undertaken by business support agencies or private-sector business groups. Additional support services, workshops and networking events were provided to help businesses (particularly small- and medium-sized enterprises) to become better placed to compete for both Olympics-related and non-Olympics-related contracts.
Our evaluation of the CompeteFor programme in Leicestershire was originally designed after initial discussions with key stakeholders. However, the initial proposal subsequently proved to be problematic for a number of reasons. First, under the strict data protection rules of the London 2012 Organising Committee of the Olympic and Paralympic Games (LOCOG), it was not possible to carry out qualitative or quantitative analysis of the successful businesses which had won the London 2012 contracts in Leicestershire at the time when the research was being conducted (names of successful bidders for 2012 contracts were declared to be confidential). Furthermore, the total number of sub-contracts won by Leicestershire businesses and the value of each contract were not revealed which, again, was a significant limitation of the data. Third, although the research team was made aware of the existence of a national longitudinal evaluation of CompeteFor, access to the three full reports (baseline, interim and final impact evaluations of the programme) was denied though data from the interim national study of CompeteFor was available in a published study which used the UK national data in the context of a study commissioned by Scottish Enterprise (Evaluation Partnership, 2011).
Thus, drawing largely on the regional statistical data collected by the East Midlands Development Agency (emda), on the analysis of key regional policy documents and on three interviews with the programme’s internal stakeholders (lasting approximately 60 minutes each), our discussion demonstrates how process tracing can be used. Those interviews sought to identify what, if any, additional policy and promotional support had been provided by virtue of the London 2012 Games, and more importantly to explore the assumptions made by the stakeholders in relation to the actions taken, and their impact on intended outcomes.
Figure 2 provides an illustration of the programme theory, the logic underpinning the programme and the actions taken to secure intermediate and final outcomes. The first element of programme theory was the contention that establishing the CompeteFor portal would increase the flow of information between buyer and supplier organisations, and thus increase the chances of Leicestershire organisations being shortlisted for contracts. The actions required to make this happen include provision of the portal, outreach work with buyer and supplier organisations to foster use of the portal as the primary source for advertising opportunities and those seeking opportunities for contracts relating to the 2012 London Olympics.
Programme and action logic model for the Leicestershire CompeteFor programme.
The second step in the causal linkage was the claim that the numbers registering on the CompeteFor website and subsequently publishing the details of their profile would be increased, if skills workshops fostered by CompeteFor, and provided by third parties (Business Links) address skills shortcomings as identified by regional analysis of business skills. Thus, at the level of action theory, analysis of skills shortages and the provision of workshops and seminars to deal with such shortcomings would result in an increased number of registered businesses proceeding to the stage of posting their profiles on the portal.
The third step in programme theory was the contention that businesses whose profiles were published on the CompeteFor portal would be more likely to be shortlisted for 2012 contracts. Here, the claim is that ‘buyers’ would not have to research the size, history, skills and track record and so on, of supplier organisations whose profiles were published on the platform, and because this information was readily available would be more likely to select organisations for short listing whose profile was known.
If we move beyond programme- and action-theory-related concerns to consider process tracing for CompeteFor, we address the following hypothesis:
H2: The staging of the Olympic Games presents a leverage opportunity for generating positive outcomes to enhance business performance and competitiveness through businesses’ engagement with a business programme (i.e. the CompeteFor programme).
As explained, although we have a problem with data availability (and, indeed, this was also a problem for the unpublished evaluation of the programme), Table 2 provides examples of tests suitable for the available data and also of tests that would have been suitable for use with other data, had we managed to capture it. In terms of the prior probability of H2, as revealed by both document analysis and key stakeholder interviews, the Olympics enabled an amalgamation that would not have otherwise been possible: the bringing together of the contract database – linked to major public- and private-sector buying organisations – and Business Link network’s training and support services. In terms of the rationales for businesses registered in the CompeteFor programme, the Olympics offered great incentives for businesses to engage with the CompeteFor programme: CompeteFor was the only official portal for the publication of London-2012-Olympic-Games-related business opportunities, providing access to at least 20 per cent of the contracts in the London 2012 supply chain. Hence, the staging of the Olympic Games provided otherwise unavailable resources as well as prepared businesses to win future mega-sporting-events-related contracts as a result of these businesses’ engagement with CompeteFor. Overall, prior confidence in H2 was high.
Assessing the evidence for H2 (the CompeteFor programme).
As Table 2 indicates, H2 passed three hoop tests (E2.2, E2.3 and E2.8) and three straw-in-the-wind tests (E2.1, E2.4 and E2.5). However, passing those tests is not particularly informative, as this only affirms the hypothesis, but does not confirm it. Registration/publication (E2.4 and E2.5) of a business’ profile is clearly not a necessary condition for it being shortlisted, as buyer organisations are not required to select organisations for shortlisting solely from the CompeteFor website. Nor, indeed, can registration/publishing of a business’ profile be regarded as a sufficient condition for shortlisting, as clearly not all businesses registered on the website were shortlisted for an Olympics-related contract.
Instead, more solid evidence is required for a smoking-gun test to evaluate the claim that registering/publishing profile data (E2.6) and/or business skills training associated with the CompeteFor system (E2.8) (provided as a result of hosting London 2012) enhanced businesses’ chances of being shortlisted for and/or of winning contracts. In this respect, H2 would be weakened if these two tests were failed. However, even if data about the performance of unregistered/unpublished businesses were available, and those organisations in the CompeteFor scheme that were registered/published did outperform those that were not, the attributional cause of the programme’s impact and the Olympics’ impact would not necessarily be assured, as an alternative explanation would have to be tested, namely that it was the business-ready organisations which chose to join the CompeteFor scheme because they were more attuned to looking for any means of improving their competitiveness. Thus, we cannot know whether organisations were more business-ready because they availed themselves of CompeteFor programme opportunities or if organisations participated in the CompeteFor programme because they were more business-ready and thus more likely to recognise the benefits of participating in the scheme.
Although it was not possible to compare the performance of those businesses registered/published on the CompeteFor site with those which were not, the interim evaluation (Evaluation Partnership, 2011) published data (see Table 3 below) showing that the first year of operation manifested an increase in the total number of supplier businesses registered on the CompeteFor web platform, an increase also in the percentage of those businesses registered which had been shortlisted and an increase in the number of contracts won by CompeteFor suppliers. However, the increase in CompeteFor registered organisations which were shortlisted for Olympic contracts between 2010 and 2011 may simply be a product of an increase in business activity around Olympic contracts as the staging of the Games approached, rather than constituting evidence of enhanced performance.
CompeteFor overarching supplier outputs at interim stage.
Source: Evaluation Partnership (2011).
Conclusion
This study sought to unpack the complexity of assessing the Olympics’ impacts through the articulation of programme theories and action theories for both sport-related and non-sport-related impacts of the two programmes. Although there was a lack of strong empirical data in both cases, the study attempted to demonstrate how process tracing could be employed to provide added value when assessing Olympic legacy claims; it also established tests suitable for evaluating the weight of the evidence as well as for confirming hypotheses.
In both programmes, the application of process-tracing logic helped first to interrogate assumptions built into the programme-theory and action-theory development processes and second to increase the rigour of explanations for assessing the Olympic Games’ impacts (in terms of improving evaluations’ internal validity and understanding of the Olympics’ causality).
We therefore argue that transparency regarding evidentiary claims and inferences is critical to assessing Olympic legacy claims because it can foster open communications between legacy-promise makers, legacy-programme operators and legacy evaluators; in turn, this process makes sure that there is a consistency in the assumptions made and actions taken. It further reflected the usefulness of adopting process tracing as a strategy for outlining the process of Olympic legacy development, in terms of legacy assumption-making, resource-allocation, action taking and outcome evaluation at the case level.
It would seem that in the field of Olympic impact/legacy analysis, a stage has been reached at which theoretical rather than more complex methodological efforts are most needed. Employing traditional result-based evaluation approaches or ad hoc objective-based evaluation for assessing Olympic impacts generated by a particular intervention will tend only to entail capturing gross impacts, and it is clearly inappropriate to attribute all emergent outputs and outcomes to the hosting of the Olympics, even though such outputs and outcomes may be generated by a programme/intervention with Olympic branding.
To move forward, in accordance with Van der Knaap’s (2004) view, we suggest that a theory-based evaluation (by integrating process tracing) offers a promising approach not only for enhancing sport policy development, in terms of its theoretical causality and measurability (i.e., an explicit outline of the causal assumptions and expectations on which policy making and measures will be based), but also for improving policy implementation (being more explicit in informing programme operators’ understanding of what has causal impact and why) and evaluation (to facilitate policy learning).
Footnotes
Declaration of conflicting interests
The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Funding
The author(s) received no financial support for the research, authorship, and/or publication of this article.
