The challenges of contamination in evaluations of childhood interventions

Abstract

Contamination in social programmes occurs when the control group either actively or passively receive some or all of the intervention intended for the treatment group. While contamination is considered a potential threat to the internal validity of an evaluation, there is relatively little evidence on the magnitude and prevalence of contamination in social programmes and the extent to which it can bias the estimates of programme effectiveness. The aim of this article is to document the challenges of contamination in a number of childhood interventions currently being evaluated in Ireland using experimental and quasi-experimental methods. The article documents the experiences of addressing contamination at the professional staff level, the parent and child level and the dissemination level. It describes the methods employed to minimize contamination and the procedures used for measuring it. Finally, the consequences of contamination for both programme implementation and the research design are discussed.

Keywords

childhood intervention contamination experimental design quasi-experimental design

Introduction

A range of childhood interventions are currently being implemented in Ireland with the goal of improving outcomes for children and their families. A key feature of these interventions is that they are subject to rigorous evaluation using either experimental or quasi-experimental methods. Adopting such designs require that the services provided to the treatment group are not received by the control or comparison groups. If this occurs, the process is referred to as contamination. Contamination occurs when the control group either actively or passively receive some or all of the intervention intended for the treatment group (Cook and Campbell, 1979). Contamination, also known as unintentional crossover, leakage (Plewis and Hurry, 1998), spill-over effects (Bloom, 2005) or diffusion (Shadish et al., 2002), may arise for multiple reasons including administrative error, deliberate subversion by programme staff, or an exchange of information between the treatment and control groups. The process of contamination differs depending on the form of the intervention and may occur at the professional staff level, the parent and child level and the dissemination level.

While contamination is considered a desirable feature of programmes operating under normal service delivery, as providing services to a small group of individuals has positive externalities for a larger group, for programmes undergoing rigorous evaluation, contamination is a threat to the internal validity of the study. Contamination is particularly undesirable in experimental or quasi-experimental evaluations as it may bias the results by reducing the magnitude of the differences in the point estimates between the treatment and control group (Torgerson, 2001). If this occurs, the true effect of the programme will be underestimated.

Despite concerns that contamination may bias the results of rigorous evaluations, there is relatively little evidence on the magnitude of contamination and its effect on biasing estimates of social programmes (Keogh-Brown et al., 2007; Watson et al., 2005), particularly those involving parents and children. The aim of this article is to document experiences of contamination in a series of evaluations of Irish childhood programmes. The interventions focus on child and family well-being and are being implemented using a variety of delivery mechanisms (e.g. individual, school, and community-based interventions). This article describes the experiences of four initiatives funded by The Atlantic Philanthropies, and/or the Department of Children and Youth Affairs (DCYA), and include the Wizards of Words (WoW) programme conducted by Barnardos, a national children’s charity, the Healthy Schools Programme (HSP) and an early childhood care and education (ECCE) programme facilitated by the Childhood Development Initiative (CDI) in west Dublin, and the Preparing for Life (PFL) programme in north Dublin conducted by the Northside Partnership. The article provides an overview of the contamination literature and describes the potential for contamination in the four childhood interventions. It also describes the methods used to limit contamination from occurring, and the procedures used for measuring it. Finally, the positive and negative consequences of contamination for both programme implementation and the research design are discussed.

Literature review

The potential for contamination is particularly high in social or educational interventions involving behavioural change as the information is more readily transferable (Cook and Campbell, 1979). However, much of the literature to date is confined to studies of clinical and medical education interventions.

A review of contamination in trials of medical education interventions found that the reported rates of contamination varied from 0–65 percent depending on the type of intervention under consideration (Keogh-Brown et al., 2007). The rates of contamination in this review were measured in multiple ways and included measures of the amount of cross-talk between the groups, whether the control group received the intervention materials or attended intervention sessions intended for the treatment group, and whether the same professionals were in contact with both treatment conditions. For example, a study of contamination within a sexual-risk reduction intervention for adolescent females, whereby the treatment group received group-based sessions on HIV prevention and the control group received group-based sessions on general health promotion, found that 73 percent of treatment participants reported some level of cross-talking with the control group in regards to sharing information about the content of the intervention sessions (Lang et al., 2009). Such contamination occurred as both groups shared transportation to the common intervention site, with multiple opportunities to interact. Similarly, a community-based healthy food intervention reported contamination rates of between 16 percent and 54 percent (Brock et al., 1997). Unintentional cross-over was measured using a ‘cross-over’ scale based on the control groups’ responses to questions regarding knowledge of the programme, material received about the programme, and whether the material and information changed their behaviour. Similarly, a number of studies by Courneya and colleagues (2003, 2004) evaluating exercise programmes for cancer patients, found contamination rates of between 22 percent and 52 percent based on physical exposure to the intervention. Conversely, an analysis of a school-based physical education intervention found no evidence of contamination between the treatment and control groups, which was explained by the intervention being provided to children of different age groups who did not play together (Labarre et al., 1994).

Despite the prevalence of contamination in such trials, there is mixed evidence on its impact on evaluation results. For example, Lang et al. (2009), discussed above, reported no difference between the contaminated group and the uncontaminated group in terms of intervention outcomes, suggesting that receiving information through the treatment group did not result in behavioural change. However, Brock et al. (1997) found that the difference between the contaminated control group and treatment group was less than the uncontaminated control group and the treatment group, suggesting that contamination did influence the results.

Concerns around minimizing contamination can often influence the evaluation design of social programmes. For example, cluster randomized trials, whereby groups of individuals (i.e. entire schools or communities) are randomized to different treatments, are considered an optimal means of reducing the likelihood of contamination as the treatment and control groups are typically not in physical or geographical contact with one another (Torgerson, 2001). A review of 158 health interventions by Keogh-Brown et al. (2007) examined whether trials that avoided contamination, by adopting a cluster randomization design, had larger effects than trials that adopted an individual randomization design, based on the assumption that contamination is less likely to occur in clustered designs. Overall, they found little difference between the two types of trials in terms of the effectiveness of the intervention, suggesting that contamination did not bias the effects of educational interventions. However, when they restricted the analysis to homogenous trials and trials of higher quality, they found some evidence that trials that adopted a cluster randomized design had larger effects. They concluded by stating that ‘the published evidence of bias caused by contamination in trials of educational interventions is thus suggestive but weak’ (Keogh-Brown et al., 2007: 25).

In regard to educational interventions, many studies cite the use of cluster randomization to circumvent contamination, despite the lack of empirical evidence on the prevalence of contamination in such interventions. A study by Craven et al. (2001) tested for the presence of contamination in a school-based intervention focusing on self-concept by comparing a within-classroom and between-classroom design. They found that students in the within-classroom control group had higher self-concept than students in the between-classroom control group, indicative of contamination.

One of the few studies to discuss contamination in the context of a parenting intervention was conducted by Stewart-Brown et al. (2004). They suggested that the initial positive effects of the programme observed at six months dissipated by the 12-month follow-up due to contamination, as qualitative interviews with the control group revealed that the research questionnaires encouraged the participants to review their parenting practices and 10 percent of the control group attended another parenting programme before the 12-month follow-up.

One of the most useful texts on contamination, by Howe et al. (2007), reports the results of a Delphi exercise to gauge expert consensus on the factors that promote and inhibit contamination in educational interventions. It reported that contamination was most likely to occur when:

participants in the treatment and control groups lived, worked or interacted closely together;

the intervention is desirable, simple and knowledge based;

the intervention is aimed at professionals rather than individuals;

the intervention uses broadcast media, audiovisuals or written information.

It reported that contamination was least likely to occur when:

participants are socially or physically separate;

interventions are complex or aimed at changing behaviours;

cluster randomization is used.

The remainder of the article reviews the risks of contamination in a series of Irish childhood intervention, described below, and the steps taken to minimize and measure it.

Description of the Irish childhood interventions

Barnardos

Barnardos was established in 1962 and works with children living in disadvantaged communities from birth to 18 whose well-being is under threat. Barnardos provides a range of services in a variety of community-based projects and centrally located services. Barnardos is being supported by The Atlantic Philanthropies in the implementation of its 2006–12 Family Support Strategy. Examples from the Wizards of Words literary programme, which is one of the key services included in the strategy, is used in this article. Wizards of Words (WoW) is a school-based literacy programme that pairs volunteers aged 55 years and older with 1st and 2nd (aged six to eight) class students who are experiencing reading difficulties. The programme is being evaluated using an experimental design over two years, whereby students are randomly assigned to the treatment group receiving WoW in addition to regular classroom instruction, or to the control group where they receive regular classroom teaching only. Due to the relatively small number of schools participating in the programme (eight schools in total, four in Dublin and four in Limerick), random allocation to treatment and control groups occurred at the individual level rather than at the classroom or school-level.¹

Childhood Development Initiative

The Childhood Development Initiative in Tallaght West is implementing a range of services to improve outcomes for children and families and improve inter-agency collaboration. Examples from two of these services – the Healthy Schools Programme (HSP) and the early years service – are used in this article. The HSP is a ‘whole-school’ programme that aims to improve health outcomes for children while enabling improved inter-agency collaboration and referrals. The programme is being evaluated using a quasi-experimental design, with five schools in the treatment group and two in the comparison group.

The early childhood care and education (ECCE) programme is a two-year daily pre-school programme that aims to ensure children are more ready for the transition to school using the HighScope curriculum. Parents of the children in the ECCE programme also participate in an evidence-based parenting programme and receive home visits from a parent/carer facilitator. The programme also provides wrap-around supports such as a dedicated speech and language therapy service. The ECCE programme was evaluated using a cluster randomized control trial design in which nine pre-school settings were randomly assigned to the treatment group and eight were assigned to the control condition.

Preparing for Life programme

Preparing for Life (PFL) is a community-based early intervention programme, conducted by the Northside Partnership. The programme aims to improve school readiness by working with parents from pregnancy until school entry. The programme is being evaluated using both an experimental and quasi-experimental design. The experimental component involves participants being randomly allocated at the individual level to either a high or low dosage treatment group. Families in both groups receive developmental toys annually, access to one year of enhanced pre-school, public health workshops, and a PFL support worker who can help them access additional services. The high treatment group also receives bi-monthly home visits from a trained mentor to support them with key parenting issues using a set of PFL developed tip-sheets and group parent training using the Triple P Positive Parenting programme. The quasi-experimental component includes an additional comparison group, who rank closely to the PFL community in terms of socioeconomic demographics, but do not receive any treatment. All three groups are followed throughout the life of the programme and are systematically assessed at eight time points between pregnancy and school entry.²

Risks of contamination in evaluations of childhood interventions and strategies for minimizing and measuring contamination

Contamination may occur through multiple mechanisms including professional staff, programme participants, or dissemination activities. Each form is discussed below and examples of the strategies adopted by the Irish childhood interventions to both minimize and measure contamination are described.

Contamination among professionals

Contamination via professionals may occur in three ways. The first is when professionals provide a partial intervention or deliver compensating activities to participants assigned to the control group; this is called compensatory equalization (Cook and Campbell, 1979). This can include providing programme materials (e.g. leaflets and tip sheets), or when the professional provides the treatment service directly to a child or family (e.g. a home visit). Professionals may provide additional benefits to vulnerable children in the control group as they may be convinced of the value of the programme and may wish to compensate such children with a partial treatment or similar alternative. Alternatively, the control group may request that they receive the treatment and staff may agree to provide it in order to avoid family disengagement from the service. Such compensatory equalization may also occur if professional staff are ethically opposed to the idea of experimental or quasi-experimental evaluations.

Compensatory equalization may thus be intentional or unintentional. For example, the potential for intentional contamination exists in the WoW reading programme, as teachers may decide to offer children in the control group additional support outside of the classroom (e.g. allocation of additional resource hours or participation in another specialist programme). Alternatively, teachers may unintentionally provide more in-classroom support to the control group to compensate them for the absence of WoW.

In order to minimize such contamination in the WoW programme, teachers were requested to nominate to the programme only those children who were not receiving additional reading support outside of the WoW programme. However, if information about a child’s receipt of additional reading support became known after the child was allocated to the treatment or control condition, the type and dosage of this additional support was noted in order to estimate its impact on outcomes. This information was elicited using a self-completion questionnaire that each participating child’s teacher completed at the beginning and end of the school year. The survey included questions about each child’s reading ability and their reading behaviour, in addition to detailed questions to gauge potential contamination, such as whether the children were receiving formal or informal learning support, how such supports were provided (i.e. in the classroom or out-of-the classroom via the teacher, special needs assistant, or a learning support teacher in small groups or one-to-one setting), and the duration and frequency of the supports. Furthermore, WoW staff maintained detailed records on each child’s reading progress and regularly liaised with the classroom teacher about each child attending WoW. This information was included in the statistical modelling used to estimate programme effects.

The potential for contamination has also been noted with respect to the Healthy Schools Programme, as the schools in the comparison group may develop more health promoting systems, processes, and practices with a view to meeting the health and well-being needs of their students. If this occurs it may result in improved health outcomes for children attending the comparison schools. While no formal monitoring of comparison school activities was conducted, comparison school staff were interviewed as part of the evaluation process regarding the health promoting programmes being implemented. The information will in turn be used to interpret the outcomes findings from the study.

The second form of contamination from professionals may occur when staff are working with both study conditions. For example, contamination may occur when exposure to new training changes the practices of professionals, such that they provide the intervention to both treatment and control group participants through their general up-skilling (Craven et al., 2001). Such training has the potential to alter their behaviours, attitudes, and skills and thus alter the way in which they work with service users, irrespective of study condition. Furthermore, professional staff may change roles within an organization during the lifetime of the evaluation. For example, a staff member may begin by providing services to the treatment group and through a role change or promotion provide services to the control group or supervise other staff working with participants in both study conditions, as occurred during the ECCE programme evaluation. Control group participants who engage with these staff may inadvertently benefit from these changed or improved working practices.

The third form of professional contamination may occur when professionals who have received training to deliver the programme have the opportunity to share information, skills, and techniques with professionals working with the control group. For example, in the ECCE intervention, two centres from the same organization participated in the evaluation with one centre being randomly allocated to provide the enhanced programme and the other centre allocated to the control group providing the standard intervention. While the programmes were discrete and were offered in centres geographically distant from each other, contamination may have occurred as there were opportunities for staff to share experiences and learning through organizational structures. Inadvertent contamination may occur as staff delivering the programme in the two centres share the organizational ethos, work to the same set of organizational best practice standards, share the same line management structure, attend meetings, and participate in organizational training and development events.

The strategy used to record and quantify such contamination was the use of research journals. ECCE evaluation team members were encouraged to be reflective when engaging with parents, children, pre-school staff and the delivery organization, and the research journals were used to record potential and actual instances of contamination. The journals were completed during and after field work activities and recorded information about implementation and programme components under three broad headings: fidelity, organization, and utilization. Observations of implementation practice under these headings were discussed with and notified to the lead researcher. The details captured in the research journals were used in the estimation of quantitative outcomes and provided added detail and an additional interpretive layer to the quantitative findings.

Contamination among programme participants: Parents and children

The likelihood of contamination may also be influenced by the way in which the control and treatment groups are assigned. Contamination may be exacerbated if the allocation is made at an individual level, particularly when the individuals are in close contact and can avail of opportunities to share experiences and learning. Contamination via parents or children occurs when those in receipt of an intervention intentionally or unintentionally share information, strategies, advice, or materials with members of the control group, thus dispersing the effect of the intervention (Lang et al., 2009). For example, PFL is a dosage experiment such that participants are randomly assigned at an individual level to a high treatment group and a low treatment group. As PFL is operating in a very small community with a population of <7000, the potential for contamination is high as some of the participants in the two treatment groups are neighbours, friends, colleagues, or even members of the same family. Contamination would result from participants in the high treatment group sharing the parenting materials/advice that they receive from their mentors with participants in the low treatment group.

A number of strategies were used to both minimize and measure contamination in the PFL evaluation. First, both the high and low treatment groups are also being compared to a ‘services as usual’ comparison group, who do not receive the PFL programme, and are geographically separated from the PFL catchment community. This group was selected using small area population statistics from the Census to rank all 322 communities in Dublin in terms of their closeness to the PFL community based on standard demographic and socio-economic characteristics. Data from the participants in the comparison community, which is located 7 km from the PFL community, are collected at the same time points as the PFL treatment groups. This additional comparison group allows us to evaluate the impact of the programme even if contamination is present within the high and low PFL treatment groups.

Several strategies were also devised to measure cross-talk and information flows between the PFL treatment and control groups. First, a series of ‘contamination’ or ‘blue-dye’ questions were included in each wave of data collection. These questions ask participants from all three groups if they have heard of particular child development phrases, and if they know what these phrases mean. All of the phrases are related to topics that only participants in the high treatment group should be aware of as the mentors discuss these issues with participants during programme delivery – for example, ‘mutual gaze’, ‘circle of security’, and ‘secure base’. Specifically, the mentors have developed ‘tip-sheets’ on these topics, which they give to the participants during the home visits. These questions can be used to measure contamination as if the low treatment group state that they know what these phrases mean and they correctly identify how to promote such behaviour, it is indicative that they may have accessed material or information intended for the high treatment group only. By comparing the responses of the high and low treatment groups to the comparison group, where contamination could not have occurred given the geographic distance, an estimate of the magnitude of contamination can be derived.

In addition, information was directly elicited from the participants regarding how many people the participants know taking part in the PFL programme, how often they meet with other PFL participants, who they discuss the programme with, and whether they share their PFL materials with anyone. While these questions will not identify contamination directly, it will provide some information on the extent of the contamination problem.

Inadvertent contamination may also occur in the WoW programme via the participating children as randomization was made at the individual level. Control group children may benefit from sharing the classroom with children who are in receipt of WoW, as the treatment children share their newly acquired skills, attitudes, or reading behaviours with their peers in the control group. In addition, teachers may react to the overall higher level of reading skills in the classroom by changing the level at which they teach, thus inadvertently affecting the control group. The reading strategies adopted by teachers during the implementation and evaluation of the programme were explored during the qualitative interviews with teachers.

Contamination by dissemination

Contamination may also occur when information about the programme is shared in advance of, or during, programme delivery. The process of dissemination prior to programme rollout, which involves engaging and informing the community about the experimental trial through community meetings, open day events or advance publicity materials, may inadvertently cause contamination by creating opportunities for discussion and raising curiosity about the research methodology, the process by which participants will be allocated, and the interventions to be tested. This in turn may bias the study as participants change their behaviour, share information, or seek services elsewhere.

At the commencement of the ECCE programme evaluation, it was intended that staff from the treatment and control groups were to be brought together to acknowledge and celebrate the start of the process. However, due to concerns about the potential for contamination through exposure to programme content and opportunities to share information between the staff groups at such an early stage in the evaluation process, it was agreed that only staff from the treatment group should attend the event. Similarly, in the PFL programme a community event was held to celebrate the recruitment of all the participants. While participants from the high and low treatment groups were invited to attend, no evaluation data was presented.

The measurement of how dissemination activities affects the behaviour of control group participants is important if the trade-off between the benefits of sharing information and its drawbacks are accepted by programme evaluators and providers. In this case, it is necessary to record the control groups attendance at dissemination activities, the type of information gained, and if and how their attitudes and behaviours have changed as a result of exposure to such information. It is also useful to determine whether or not they have sought out participation in other similar programmes on the basis of information received. In the PFL evaluation, the presence of an external control group, who are not included in any of the dissemination activities, will help to assess the impact of such activities on the behaviour of participants in the low treatment group.

Contamination by dissemination may also occur as a result of sharing incomplete or interim study findings. Rigorous evaluations often take a long time to complete and sharing information and communicating about progress and the achievement of milestones is an important means of keeping stakeholders engaged with the evaluation process. However, sharing interim findings that explicitly identifies differences between the control and treatment groups may result in changes in behaviours or attitudes in either group or encourage participants to seek alternative services. For example, if negative or null interim findings are reported to the programme implementers, it may encourage them to make programmatic changes mid-way through the study. This could lead to additional effort being invested or may de-motivate service providers. Similar effects may occur when positive findings are reported and service providers may scale back their effort in response. Thus, evaluators must carefully consider the effects of releasing interim findings, and, where possible, releasing such data should be minimized and controlled. For example, in the ECCE evaluation, it was agreed that in order to be inclusive and keep participants informed of progress, interim results were shared with staff, parents, and children. However, in order to minimize the potential for contamination, intervention and control labels were not used in the presentation of results. Similarly, in PFL, which is taking place over six years, interim results are shared with key stakeholders in the community. While the interim results are not fed back to the participants directly, they receive regular newsletters with information on upcoming events, etc.

In some instances, evaluators and programme providers may agree that the potential for contamination is outweighed by the benefits of engaging in dissemination activities that promote initial and ongoing participation in the programme. This process may reduce concerns or suspicions about the use of experimental methodologies and may promote transparency and ownership of the evaluation process.

Discussion

A review of the experiences of the Irish childhood interventions clearly demonstrates the challenges faced in conducting social programme evaluation. The article highlights the importance of conducting in-depth studies into the presence of contamination as the magnitude of contamination has several implications for the evaluation results, policy recommendations, and implementation practices. First, the reliability of results based on a contaminated control group may be questionable, particularly if the level of contamination was such that it improved the outcomes of the control group to similar levels of the treatment group. The failure to detect a significant programme effect limits the range of policy recommendations that can be made and may affect future funding and programme implementation. By conducting a thorough study of the extent of contamination it is possible to better interpret the evaluation results. For example, rather than concluding that a particular programme is not effective, the conclusion is modified to note that a programme effect cannot be detected due to contamination and that such contamination suggests that the programme is modifying the behaviour of both the treatment and control groups and therefore may be effective.

Second, the presence of contamination in social experiments, while often discussed as a potential threat to internal validity, is rarely documented. Despite the dearth of evidence, the need to minimize contamination is frequently cited as an explanation for choosing a cluster randomization design over individual level randomization (Bloom et al., 2007; Donner and Klar, 2000; Raudenbush, 1997). Investigating the existence and impact of contamination is therefore important, particularly as the advantages of using cluster randomization to minimize contamination may be offset by the drawbacks associated with these designs, such as the need for an increased sample size, which increases the cost and duration of the evaluation (Torgerson, 2001). In addition, statistical power is greater when randomization occurs at the individual level, as the standard errors of programme effects are typically larger when cluster randomization is used (Cornfield, 1978; Schochet, 2008). A recent study by Rhoades (2011) finds that an individual randomization design is always preferable as it has more power than a cluster design, even when contamination reduces the effect size by between 10–60 percent.

This article has described a number of strategies that can be utilized to measure contamination. Direct measures, such as asking participants (either professionals or families) whether they know anyone participating in the programme, whether they share the materials they receive, and whether they talk about the programme with others, can be collected as part of routine assessments. If available, information on the participants’ home or work addresses can be used to gauge the physical distance between the participants. In addition, while not currently used in any of the evaluations discussed here, a social network analysis could also be used to measure relationship and information flows between participants (Hawe and Ghali, 2008; Wasserman and Faust, 1984).

A more precise method of measuring contamination is to incorporate ‘blue-dye’ questions into standardized questionnaires. As described above in the case of PFL, these questions can be used to determine whether the control group is aware of certain elements of the programme or engage in certain behaviours promoted by the programme. If contamination occurred, the difference in responses of the treatment and control groups on these questions would represent the level of programme leakage.

Implementation analyses, which are used to determine whether the intervention is being delivered as intended (Durlak and Dupre, 2008), can also be used to measure contamination. Implementation analyses involve multiple components including direct observation of service delivery to measure programme fidelity, the recording of all intervention activities to measure programme dosage, and qualitative assessments with programme staff and participants to assess programme satisfaction. The presence of contamination can be elicited through each of these components by comparing the treatment and control groups on each domain. Finally, as discussed above, another method for tracking contamination is through recording any incidences of contamination in a research journal. While many of these methods will not allow you to quantify the magnitude of contamination directly, they provide a useful indicator that can be used to interpret the results of the evaluation.

Another issue associated with contamination in experimental trials, which has not been discussed in this study, is blinding. Double blinding is possible in clinical trials such that the treatment condition of participants is unknown to both participants and implementation staff. In social experiments, however, blinding is often not possible as the participants are aware that they are receiving additional services or not and the implementers are aware that they are delivering services to one group but not the other. Contamination may thus occur if participants change their behaviour as a result of their treatment condition. For example, the control group may attempt to access the services elsewhere. In all the interventions discussed here, neither participants nor implementers are blinded to the treatment condition; however, efforts are made to monitor such activities of the control group. In addition, the evaluators are blinded to ensure that the collection and interpretation of the evaluation data is not biased; however, in certain cases, participants may inadvertently reveal their treatment condition to the evaluators.

While contamination is generally considered a negative outcome within experimental evaluations, within real world settings contamination can be considered a positive externality. Positive externalities occur when the benefits of the programme are transferred to individuals who were not the intended recipients. Thus, contamination is a positive outcome for policymakers and practitioners wishing to expand the benefits of the programme to additional recipients without the additional cost. Measuring such levels of diffusion can help inform the likely success of programme roll-out and may influence scale-up decisions. If contamination is present and the programme demonstrated positive effects, using the documented modes of contamination can help replicate the results on a wider scale. For example, providing opportunities for cross-talk, such as joint meetings and events, may spread the benefits of the intervention to a wider audience. Similarly, holding open days and sharing results may encourage individuals who are not receiving the programme to seek out information about the programme.

Contamination is thus a double-sided sword, often feared by evaluators and embraced by policymakers and practitioners. Studies of the magnitude of contamination are essential in order to select the most appropriate evaluation design on the one hand and to promote programme roll-out on the other hand, by applying the optimal contamination processes to promote successful contagion of the intervention.

Footnotes

Funding

This research received no specific grant from any funding agency in the public, commercial or not-for-profit sectors.

Notes

Orla Doyle is a lecturer in the UCD School of Economics and a research fellow at the UCD Geary Institute.

Claire Hickey is the senior research manager with Barnardos, Ireland’s leading children’s charity; she is responsible for implementing and managing Barnardos research and evaluation programme nationally.

References

Bloom

(2005) Randomizing groups to evaluate place-based programs. In: Bloom

(ed.) Learning More From Social Experiments: Evolving Analytic Approaches. New York: Russell Sage Foundation, 115–72.

Bloom

Richburg-Hayes

Black

(2007) Using covariates to improve precision: empirical guidelines for studies that randomize schools to measure the impacts of educational interventions. Educational Evaluation and Policy Analysis 29: 30–59.

Brock

Mauney

Kalsbeek

(1997) Unintentional crossover effects in community intervention studies: evidence from a nutrition survey aimed at African Americans. American Statistical Association, Proceedings of the Section on Survey Research Methods, 667–72.

Cook

Campbell

(1979) Quasi-experimentation: Design and Analysis Issues for Field Settings. Boston, MA: Houghton Mifflin.

Cornfield

(1978) Randomization by group: a formal analysis. American Journal of Epidemiology 108: 100–2.

Courneya

Freidenreich

Quinney

Fields

ALA

Jones

Fairey

(2004) Predictors of adherence and contamination in a randomized trial of exercise in colorectal cancer survivors. Psycho-Oncology 13: 857–66.

Courneya

Friedenreich

Sela

Quinney

Rhodes

Handman

(2003) The group psychotherapy and home-based physical exercise (group–hope) trial in cancer survivors: physical fitness and quality of life outcomes. Psycho-Oncology 12: 357–74.

Craven

March

Debus

Jayasinghe

(2001) Diffusion effects: control group contamination threats of the validity of teacher-administered interventions. Journal of Educational Psychology 93(3): 639–45.

Donner

Klar

(2000) Design and Analysis of Cluster Randomization Trials in Health Research. New York: Oxford University Press.

10.

Durlak

Dupre

(2008) Implementation matters: a review of the research on the influence of implementation on program outcomes and factors affecting implementation. American Journal of Community Psychology 41: 327–50.

11.

Hawe

Ghali

(2008) Use of social network analysis to map the social relationships of staff and teachers at school. Health Education Research 23(1): 62–9.

12.

Howe

Keogh-Brown

Miles

Bachmann

(2007) Expert consensus on contamination in educational trials elicited by a Delphi exercise. Medical Education 41: 196–204.

13.

Keogh-Brown

Bachmann

Shepstone

Hewitt

Howe

Ramsay

. (2007) Contamination in trials of educational interventions. Health Technology Assessment 11(43).

14.

Labarre

Jequier

Shephard

Lavallee

Rajic

(1994) Testing for inter-group contamination in a controlled longitudinal study of added physical education. Journal of Sports Medicine and Physical Fitness 34: 403–6.

15.

Lang

DiClemente

Hardin

Crosby

Salazar

Hertzberg

(2009) Threats of cross-contamination on effects of a sexual risk reduction intervention: fact or fiction. Prevention Science 10: 270–5.

16.

Plewis

Hurry

(1998) A multilevel perspective on the design and analysis of intervention studies. Educational Research and Evaluation 4: 13–26.

17.

Raudenbush

(1997) Statistical analysis and optimal design for cluster randomization trials. Psychological Methods 2: 173–85.

18.

Rhoades

(2011) The implications of ‘contamination’ for experimental design in education. Journal of Educational and Behavioral Statistics 36(1): 76–104.

19.

Schochet

(2008) Statistical power for random assignment evaluations of education programs. Journal of Educational and Behavioral Statistics 33: 62–87.

20.

Shadish

Cook

Campbell

(2002) Experimental and Quasi Experimental Designs for Generalized Causal Inference. Boston, MA: Houghton Mifflin.

21.

Stewart-Brown

Patterson

Mockford

Barlow

Klimes

Pyper

(2004) Impact of a general practice based group parenting programme: quantitative and qualitative results from a control trial at 12 months. Archives of Disease in Childhood 89: 519–25.

22.

Torgerson

(2001) Contamination in trials: is cluster randomization the answer? British Medical Journal 322: 355–7.

23.

Wasserman

Faust

(1984) Social Network Analysis: Methods and Applications. New York: Cambridge University Press.

24.

Watson

Kendrick

Woods

Dewey

(2005) Measuring contamination in an injury prevention randomized controlled trial. International Journal of Injury Control and Safety Promotion 12(3): 197–200.