A cross-validation trial of an Internet-based prevention program for alcohol and cannabis: Preliminary results from a cluster randomised controlled trial

Abstract

Objective:

Replication is an important step in evaluating evidence-based preventive interventions and is crucial for establishing the generalizability and wider impact of a program. Despite this, few replications have occurred in the prevention science field. This study aims to fill this gap by conducting a cross-validation trial of the Climate Schools: Alcohol and Cannabis course, an Internet-based prevention program, among a new cohort of Australian students.

Method:

A cluster randomized controlled trial was conducted among 1103 students (M_age: 13.25 years) from 13 schools in Australia in 2012. Six schools received the Climate Schools course and 7 schools were randomized to a control group (health education as usual). All students completed a self-report survey at baseline and immediately post-intervention. Mixed-effects regressions were conducted for all outcome variables. Outcomes assessed included alcohol and cannabis use, knowledge and intentions to use these substances.

Results:

Compared to the control group, immediately post-intervention the intervention group reported significantly greater alcohol (d = 0.67) and cannabis knowledge (d = 0.72), were less likely to have consumed any alcohol (even a sip or taste) in the past 6 months (odds ratio = 0.69) and were less likely to intend on using alcohol in the future (odds ratio = 0.62). However, there were no effects for binge drinking, cannabis use or intentions to use cannabis.

Conclusion:

These preliminary results provide some support for the Internet-based Climate Schools: Alcohol and Cannabis course as a feasible way of delivering alcohol and cannabis prevention. Intervention effects for alcohol and cannabis knowledge were consistent with results from the original trial; however, analyses of longer-term follow-up data are needed to provide a clearer indication of the efficacy of the intervention, particularly in relation to behavioral changes.

Keywords

Alcohol cannabis prevention school-based Internet replication

In Australia, the National Alcohol Guidelines recommend that persons under 18 years of age abstain from alcohol and that for young people aged 15−17 years, the safest option is to delay the initiation of drinking for as long as possible (National Health and Medical Research Council, 2009). Despite these recommendations, and a legal drinking age in Australia of 18 years, 51% of students (aged: 12–17 years) report consuming alcohol in the past year (White and Bariola, 2012). Similarly, although cannabis is illegal in Australia, in 2011 15% of 12- to 17- year-olds had used cannabis in their lifetime (White and Bariola, 2012). These statistics are concerning in light of the significant costs and harms associated with alcohol and cannabis use (Collins and Lapsley, 2008; Manning et al., 2013). Given that early onset of alcohol and other drug (AOD) use in adolescence is associated with later substance use and mental health problems in adulthood (Chen et al., 2009), it is critical to deliver prevention before initiation to AOD use occurs.

Delivering prevention for AOD in a school setting is ideal as it is a place where students spend a large percentage of their lives and a large universal audience can be reached (Mihalic et al., 2008). However, existing school-based prevention programs have typically only produced small effects, especially in terms of reducing AOD use (Botvin and Griffin, 2007; Foxcroft and Tsertsvadze, 2011). This is likely due to factors that impact program implementation, which in turn can affect program outcomes (Durlak and DuPre, 2008; Elliott and Mihalic, 2004). For example, schools typically lack adequate resources to deliver AOD education (Bumbarger and Perkins, 2008), and teachers often make adaptations to the content and delivery of programs, which can detract from the intervention (Dusenbury et al., 2005; Ennett et al., 2003). Therefore, it has been critical for researchers to devise new ways of delivering school-based AOD prevention that can be implemented with ease and fidelity.

Interventions facilitated by the Internet have the potential to overcome implementation obstacles and offer a number of advantages over traditional school-based prevention programs. Online programs typically offer increased feasibility of use as professionals are not required for their delivery and teachers typically need little training (Marsch et al., 2007; Schinke et al., 2004). Second, the fact that Internet interventions consist of pre-programmed content means that they can often be implemented with a higher degree of fidelity, as teachers are unable to make adaptations to core program components (Backer, 2001; Pankratz et al., 2006). Finally, the ability of online mediums to incorporate audio–visual elements and to provide tailored messages creates an interactive environment which can foster higher student engagement (Bennett and Glasgow, 2009).

In an attempt to overcome implementation barriers commonly encountered by school-based programs, the Internet-based Climate Schools: Alcohol and Cannabis course was developed. The course utilizes interactive online cartoons to engage students and is centered on a social influence (Botvin, 2000) and harm-minimization approach. It aims to provide students with information required to minimize harms associated with alcohol and cannabis use, challenge perceptions of peer drug use and build resistance skills. The course has been evaluated previously in a cluster randomized controlled trial (RCT) (n = 764 Year 8 students, M_age = 13.08 years, 60% male) in 10 schools in Sydney, Australia (Newton et al., 2009, 2010). In this trial, the course was found to be more effective than drug education as usual in increasing alcohol (d = 0.76) and cannabis (d = 0.61) knowledge, and decreasing average weekly alcohol use (d = 0.38) and the frequency of binge drinking (d = 0.17) up to a 12-month follow-up. In addition, the program was also found to significantly reduce the frequency of cannabis use (d = 0.19) at a 6-month follow-up. Although these results are promising, an important next step is to evaluate the program again to see if the effects can be replicated.

Replication is a critical step in evaluating any evidence-based preventive intervention (Valentine et al., 2011) and has been stipulated as a key criterion for efficacy by the Society for Prevention Research in their Standards of Evidence (Flay et al., 2005). Replication helps to rule out the possibility that findings from an initial trial were due to chance (Flay et al., 2005) and to determine whether a program can be generalized to new settings or make an impact on a larger scale (Elliott and Mihalic, 2004). Despite the clear need for cross-validation, few replications have been conducted in the prevention science field (Aos et al., 2011), and there have been no replications of an Internet-based AOD prevention program (Champion et al., 2013). The present study aims to fill this gap by conducting a cross-validation trial of the Climate Schools: Alcohol and Cannabis course among a new cohort of Australian students.

Methods

Design

A cluster RCT was conducted in secondary schools in Australia as part of a larger trial of an integrated universal and selective prevention program for adolescents, known as the Climate and Preventure (CAP) Study (Newton et al., 2012). The CAP Study sample consisted of 2268 students from 27 schools, which were randomly allocated to one of four trial groups. Blocked randomization was conducted by an external researcher using the online program Research Randomiser (www.randomiser.org). To cross-validate the Climate Schools: Alcohol and Cannabis course, only two groups were included in the present study: the Climate group (receiving the Climate Schools: Alcohol and Cannabis course) and the Control group (health education as usual). All students completed a self-report questionnaire at baseline and immediately post-intervention. All aspects of this study were approved by the University of New South Wales Human Research Ethics Committee (HREC 11274), and the trial is registered with the Australian New Zealand Clinical Trials Registry (ACTRN12612000026820).

Participants and procedure

A total of 14 schools were randomly allocated to either the Climate Schools intervention (n = 7) or a control group (n = 7). Consent forms were sent home to parents of all Year 8 students at participating schools. Some schools (n = 10) required passive parental consent, while students at other schools (n = 4) needed active consent due to ethical requirements. One school declined to implement the intervention due to insufficient time and were withdrawn from the study prior to baseline assessment. The final sample at baseline consisted of 1103 students (n = 576 Intervention; n = 527 Control) from 13 schools.

The intervention group

Intervention schools implemented the Climate Schools: Alcohol and Cannabis course with their Year 8 students during Personal Development, Health and Physical Education (PDHPE) classes. The course consists of the six-lesson Alcohol module, delivered in Term 1, and the six-lesson Alcohol and Cannabis module, delivered approximately 6 months later. Each lesson consists of a 20-minute online cartoon component completed individually by students, followed by teacher-led activities. Teachers were provided with a selection of pre-planned activities per lesson, including discussions, role-plays and worksheets, and were able to choose which activities to implement to best meet the needs of their class. Details of the content contained in each lesson have been described previously (Newton et al., 2010). All course materials, including the cartoons, implementation guidelines, lesson summaries and activities were accessed online via www.capstudy.org.au using secure login details, and teachers were also provided with hard-copy program manuals.

The control group

The control schools received their PDHPE lessons as usual over the course of the year, including drug education. Teachers were asked to provide details about the amount and format of any drug education they delivered to their Year 8 students.

Measures

Students completed the survey in class at baseline and immediately post-intervention, which occurred approximately 6 months after baseline (see Table 1). A total of 12 schools completed the survey online, and 1 school completed the survey via paper-and-pencil. Data were linked over time using unique identification codes to ensure confidentiality was maintained.

Table 1.

Assessment and intervention timeline.

	Baseline survey	Climate schools: Alcohol module	Climate schools: Alcohol and Cannabis module	Post-intervention F/U survey
Time^a	Term 1, February–April 2012	Term 1, February–April 2012	Term 3, July–September 2012	Terms 3 and 4, August–December 2012
Grade	Year 8	Year 8	Year 8	Year 8
Control^b	✓	✗	✗	✓
Intervention	✓	✓	✓	✓

The Australian school year runs from late January to late December.

The timing in which control schools delivered their drug education varied by school, ranging from April to December.

Demographics

Demographic information gathered included students’ age, gender, country of birth and self-reported academic grades.

Alcohol use

Alcohol use was assessed via two items. Items measured any alcohol use (including a sip or taste) and the frequency of binge drinking (five or more standard drinks on one occasion) in the past 6 months. The measure of binge drinking is based on the Australian National Health and Medical Research Council’s (2009) guideline for ‘reducing the risk of injury on a single occasion of drinking’. Due to very low cell counts for the higher use categories, responses were collapsed to create a binary variable representing the frequency of binge drinking (0 = never binged, 1 = binged once or more).

Cannabis use

A single item, adapted from the 2010 National Drug Strategy Household Survey (NDSHS) (Australian Institute of Health and Welfare [AIHW], 2008), assessed cannabis use in the past 6 months (Yes/No).

Alcohol and cannabis knowledge

Alcohol and cannabis knowledge were each assessed using 16-item scales that had been used in previous evaluations of the Climate Schools programs (Newton et al., 2010; Vogl et al., 2009). The scales assessed knowledge in relation to physical and mental health problems, legal consequences and information required to minimize harms. Students were asked to respond ‘true’, ‘false’ or ‘don’t know’ to each statement. Total scores ranged from 0 to 16, with higher scores indicating higher knowledge. Both the Alcohol and Cannabis Knowledge scales demonstrated acceptable reliability in the original study (alcohol, α = 0.61; cannabis, α = 0.77) and in the present trial (alcohol, α = 0.75; cannabis, α = 0.86).

Intentions to use alcohol and cannabis

Two similar items were used to assess future intentions to use alcohol and cannabis: ‘Please rate how likely you think it is that you will try (alcohol/cannabis) at any time in the future’. Responses were coded ‘very unlikely, unlikely, unsure = 0’ or ‘likely, very likely = 1’.

Program evaluation and implementation fidelity

Upon completion of the intervention, students and teachers were asked to complete an evaluation questionnaire about the program. To monitor adherence, teachers were asked to complete a logbook indicating which lessons and activities they completed with their class and any factors that may have disrupted delivery of program.

Statistical analyses

Statistical analyses were conducted using Stata 13 and IBM SPSS Statistics 22. Baseline equivalence between the control and intervention was examined using chi-square tests for categorical variables, one-way analysis of variance (ANOVA) for continuous outcomes and Mann–Whitney U for nonnormally distributed variables. Intraclass correlation coefficients (ICCs) were calculated for all outcomes to determine the extent of clustering between versus within schools and are presented in Table 3. All students with baseline data were included in the analyses irrespective of the number of intervention lessons they completed. All analyses used mixed-effects regression models to take into account clustering of data at the school level. Multilevel linear regression models compared the trial groups on continuous outcomes (knowledge), and logistic regressions were conducted for categorical outcomes (alcohol use, cannabis use, binge drinking and intentions). Cohen’s d was calculated from raw means and standard deviations to provide an index of effect size for continuous outcomes, and odds ratios (ORs) with 95% confidence intervals (CI) are reported for binary outcomes. All analyses were adjusted for baseline scores on the outcome variable and included a variable representing trial group (intervention or control). Gender was included as a covariate in all analyses as research indicates that males and females tend to exhibit differing levels of substance use during adolescence (Kloos et al., 2009). Academic grades were included as a covariate to adjust for differences between the groups on grades at baseline, and in light of evidence that indicates that poor academic achievement is associated with greater AOD use (Henry, 2010).

Missing data

Primary analyses were based on complete case data; thus, a series of sensitivity analyses examined the impact of response attrition on the results. Analyses were repeated on 100 datasets imputed using multiple imputation by chained equations (Royston, 2009). Multiple imputation is a recommended approach for handling missing data (Sterne et al., 2009) and assumes data are missing at random (MAR) conditional on the variables in the imputation model. To ensure the plausibility of this assumption, the imputation model incorporated auxiliary demographic, mental health and substance use variables predictive of incomplete outcome variables and/or missingness (full list available on request). Each data-set entailed 20 cycles of regression switching, and predictive mean matching was used for variables that were not normally distributed. Estimates were combined according to Rubin’s rules use the ‘mi estimate’ command in Stata.

Results

A total of 1103 students completed the survey at baseline (M_age = 13.3 years, standard deviation [SD] = 0.47; 65% female, 88% Australian), and 880 (80%) responses were received post-intervention. Figure 1 provides a breakdown of participation and survey responses over time.

Figure 1.

CONSORT flow chart of recruitment and survey completion.

Baseline equivalence

At baseline, there were some differences between the intervention and control groups. Specifically, students in the control group reported significantly higher academic grades (χ²(1, N = 1098) = 12.15, p = 0.02), greater alcohol-related knowledge (F(1, 1101) = 20.07, p < 0.001) and greater intentions to use alcohol in the future (χ²(1, N = 1082) = 24.19, p < 0.001) than those in the intervention group.

Attrition analyses

The overall attrition rate was 20% (17% for the control group, 24% for the intervention group). Attrition analyses were conducted to determine whether students that were only present at baseline differed from those who were followed-up post-intervention. Reasons for attrition included students being absent from school on the day of the survey, changing schools or moving away, failing to remember their username and password to complete the survey online or using the incorrect code to complete the survey via paper-and-pencil. To maximize follow-up rates, students that were absent for the survey were contacted by the researchers and reminded to complete the survey. Results revealed that students who were not followed up at post-intervention were more likely to be male (OR = 2.15, 95% CI = [1.60, 2.90]), reported significantly lower knowledge about alcohol (F(1, 1101) = 7.59, p = 0.01) and cannabis (F(1, 1088) = 6.24, p = 0.01) and were more likely to intend to use cannabis in the future (OR = 2.85, 95% CI = [1.30, 6.24]). However, students that were not followed-up post-intervention were significantly less likely to report intentions to use alcohol in the future (OR = 0.56, 95% CI = [0.42, 0.76]), compared to those that were present at both assessments.

Intervention effects

Table 2 reports the descriptive statistics for outcome measures at baseline and post-intervention for both groups. Table 3 presents results from the multilevel regression analyses of the nonimputed data.

Table 2.

Descriptive statistics for outcome variables at baseline and post-intervention, by group.

Outcome measure	Group	Baseline	Post-intervention
Alcohol outcomes
Any alcohol (even a sip or taste) in past 6 months (%)	INT	37.53	36.82
	CO	34.92	41.04
Frequency of binge drinking (%)	INT	4.02	7.84
	CO	3.40	6.12
Alcohol knowledge (M, SD)	INT	7.56 (2.97)	10.09 (3.33)
	CO	8.17 (2.61)	8.20 (3.07)
Alcohol intentions (% likely)	INT	47.80	48.00
	CO	60.90	62.20
Cannabis outcomes
Cannabis use in past 6 months (%)	INT	5.45	12.29
	CO	6.74	8.55
Cannabis knowledge (M, SD)	INT	6.78 (3.85)	9.95 (4.44)
	CO	6.75 (3.85)	7.14 (3.99)
Cannabis intentions (% likely)	INT	1.20	3.45
	CO	2.10	4.87

INT: Intervention group; CO: Control group; SD: standard deviation.

The statistics in this table are based on the participants included in the multilevel regression analyses.

Table 3.

Results from multilevel regression models.

Multilevel linear regression models^a
Post-intervention outcome	n	B	SE	95% CI	p-value	ICC
Alcohol knowledge	867	1.98	0.57	0.89–3.08	<0.001	0.07
Cannabis knowledge	835	2.39	0.87	0.67–4.09	0.01	0.12
Multilevel logistic regression models
Post-intervention outcome	n	OR	SE	95% CI	p-value	ICC
Alcohol use (taste or sip) past 6 months^c	860	0.69	0.12	0.50–0.96	0.03	0.01
Frequency of binge drinking	860	1.13	0.59	0.41–3.15	0.81	0.00
Intentions to use alcohol	825	0.62	0.13	0.42–0.93	0.02	0.03
Cannabis use past 6 months	832	1.54	0.41	0.92–2.59	0.10	0.02
Intentions to use cannabis	825	0.64	0.38	0.20–2.05	0.46	0.01

SE: standard error; CI: confidence intervals; OR: odds ratios; ICC: intraclass correlation coefficient.

All models account for baseline scores on outcome variables, gender and academic grades.

Adjustment for missing data weakened the intervention effect for the any alcohol use (even a sip or taste) in the past six months variable (OR=0.76, p=.16)

Any alcohol use in the past 6 months (including even a sip or taste)

Immediately post-intervention, there was a significant difference between the groups in terms of any alcohol use in the past 6 months. Specifically, students in the intervention group were less likely to report having consumed alcohol (even a sip or taste), compared to those in the control group (OR = 0.69, 95% CI = [0.50, 0.96], p = 0.03).

Frequency of binge drinking

The proportion of students that reported binge drinking in the past 6 months was low for both groups at baseline and post-intervention. A logistic regression revealed no significant difference between the intervention and control groups immediately post-intervention (OR = 1.13, 95% CI = [0.41, 3.15], p = 0.81).

Alcohol knowledge

Alcohol knowledge increased from baseline to post-intervention for both groups; however, results revealed a significant difference between the groups at post-intervention. Specifically, the intervention group reported significantly greater alcohol knowledge than the control group (β = 1.98, 95% CI = [0.89, 3.08], p < 0.001, d = 0.67).

Intentions to use alcohol

The proportion of students that reported being likely to use alcohol in the future increased from baseline to post-intervention for both groups. At post-intervention, there was a significant difference between groups, with students in the intervention group being less likely to intend on using alcohol in the future than the control group (OR = 0.62, 95% CI = [0.42, 0.93], p = 0.02).

Cannabis use in the past 6 months

Cannabis use in the past 6 months increased from baseline to post-intervention for both groups, however, reported use was low. Although cannabis use appeared to be greater in the intervention group than the control group immediately post-intervention, the odds of using cannabis did not significantly differ between groups (OR = 1.54, 95% CI = [0.92, 2.59], p = 0.10).

Cannabis knowledge

Cannabis knowledge increased over time for both the intervention and control groups; however, there was a significant difference between the groups immediately post-intervention. Specifically, the intervention group reported significantly greater cannabis knowledge than the control group (β = 2.39, 95% CI = [0.67, 4.09], p = 0.01, d = 0.72).

Intentions to use cannabis

The proportion of students in both groups that reported future intentions to use cannabis was very low at both assessment occasions. At post-intervention, there was no significant difference between the groups in terms of their likelihood of using cannabis (OR = 0.64, 95% CI = [0.20, 2.05], p = 0.46).

Missing data

Sensitivity analyses examined robustness of the complete-case analyses using datasets with imputed missing data. Results from these sensitivity analyses (see supplementary table, available online) confirmed the complete case pattern of results. Adjustment for missing data weakened the intervention effect for any alcohol use (even a sip or taste) in the past 6 months variable (OR = 0.76, p = 0.16); otherwise, there was no material difference compared to results derived from the complete-case analyses.

Teacher and student evaluations of the Climate Schools course

Of the 14 teachers that provided evaluation data, the majority rated the program positively, with 85% reporting that it was better than other programs and 92% indicating that they would recommend it to others. A sample of students (n = 195) also provided feedback about the intervention. Students evaluated the program favorably, with over 90% indicating that it was an enjoyable way of learning PDHPE theory and that they planned to use the information they had learnt in the program in their own lives.

Program delivery and implementation fidelity

A total of 23 teachers from seven schools completed the fidelity logbooks. Completion rates reported by teachers for each lesson ranged from 87% to 100% for the Alcohol module and from 69% to 92% for the Alcohol and Cannabis module. Teachers varied widely in which activities, and how many activities, they completed with their class for each lesson. All teachers reported delivering at least one activity for the Alcohol module and for the Alcohol and Cannabis module.

Control group drug education

All control schools reported implementing some form of universal AOD education with their students. The number of lessons varied between schools (ranging from 2 to 10), and the average length of each lesson was 62 minutes. More than half of teachers (57%) reported using computers or the Internet to teach AOD education topics. The main content areas covered included the effects of alcohol, laws, decision-making and influences of peers and the media.

Discussion

The aim of this study was to cross-validate the Climate Schools: Alcohol and Cannabis course to determine the efficacy and generalizability of the program. Results indicate that the program was effective in increasing alcohol and cannabis knowledge, and there was some evidence that the intervention reduced students’ intentions to use alcohol and delayed the use of any alcohol (including even a sip or taste). However, the module did not have an impact on binge drinking, cannabis use in the past 6 months or intentions to use cannabis.

Given that this study is a cross-validation trial, it is important to compare the present results with findings from the original evaluation of the Climate Schools: Alcohol and Cannabis course (Newton et al., 2010). Intervention effects for alcohol and cannabis knowledge were consistent with results from the original RCT, in terms of both significance and magnitude of effect, as well as other school-based AOD prevention programs (McBride et al., 2004; Midford et al., 2014). The increased knowledge demonstrated by intervention students provides evidence that young people can learn harm-minimization information. This finding is also important as changes in knowledge have been cited as an important intermediary step in changing AOD behavior (Midford et al., 2014).

In terms of alcohol use, there was some evidence that immediately post-intervention students in the intervention group were less likely to have drunk any alcohol in the past 6 months than those in the control group. However, this finding should be interpreted cautiously as sensitivity analyses with imputed data indicated the size of this effect was over-estimated by the complete-case analyses. Nonetheless, the potential of the Climate Schools intervention to delay students from consuming any alcohol, including even a sip or taste, has implications given the association between early initiation to alcohol use and later dependence in adulthood (Hingson et al., 2006). It will be important to examine these effects at subsequent survey occasions to see if these effects strengthen over time. The post-intervention effect for alcohol use found in the present study was not observed in the original RCT; however, this is likely explained by differences in the measurement of outcomes, which limits the comparability. That is, the measure of alcohol use adopted by Newton et al. (2010) referred to average weekly alcohol consumption, whereas the present trial assessed ‘any alcohol (even a sip or taste) in the past 6 months’.

The Climate Schools program also appeared to have an effect on students’ intentions to use alcohol in the future, with the intervention group being significantly less likely than the control group to intend on using alcohol at post-intervention, after taking into account baseline intention scores. This is an encouraging result as intentions to use alcohol have been shown to predict later alcohol use (Andrews et al., 2003). Although the intervention did not have a significant effect on cannabis intentions, the fact that the vast majority of students did not intend to use cannabis is positive in of itself, especially given the association between intentions and later cannabis use (Conner and McMillan, 1999). Unfortunately, the original trial did not assess intentions to use alcohol or cannabis, so comparisons cannot be made with the present findings.

There were no significant intervention effects for binge drinking and cannabis use at post-intervention, which is consistent with the results from the original trial of the Climate Schools course. Although cannabis use appeared to increase for both the intervention and control groups from baseline to post-intervention, importantly, there were no significant differences between the groups at post-intervention. In addition, the proportion of intervention students in this study that had used cannabis at baseline (5.4%) and post-intervention (12.3%) were similar to prevalence estimates reported in the Australian Secondary Students’ Alcohol and Drug (ASSAD) Survey (White and Bariola, 2012) for 13-year-olds (6.2%) and 14-year-olds (11.1%), indicating that it is common for the prevalence of cannabis use to increase with age. Although it is important to examine the immediate effects of the intervention, it is possible that the intervention did not produce effects on binge drinking and cannabis use due to the timing of the post-intervention survey. At both assessment occasions, students were aged approximately 13–14 years, and therefore the vast majority were not involved in harmful alcohol or cannabis use at this stage in their lives. In addition, by post-intervention students in the intervention group may not have had time to apply the preventive skills they learnt in the program to their own lives. As students age, there is likely to be greater experimental use of alcohol and cannabis and greater opportunity for students to utilize the skills in real-world situations. This may produce greater intervention effects at longer-term follow-ups, which is consistent with results from the original RCT (Newton et al., 2010). In particular, it will be important to assess whether the early effects of the intervention for alcohol and cannabis knowledge in the present study are translated into behavioral changes at later time points. Longer-term follow-up data are currently being collected as part of the CAP Study that will enable such analyses, and results will be published in 2016.

The present results should be considered in light of a number of limitations. Although most teachers indicated that it was easy to access computers to implement the intervention, some teachers did report difficulties however not to the degree that implementation was compromised. Recent Australian initiatives in which students are encouraged to bring their own laptop or tablet to school are likely to improve the feasibility of using Internet-based programs at school.

Second, attrition analyses revealed that students who were not followed-up at post-intervention were generally a higher risk sample than those that were retained after baseline. The attrition of high-risk students has the potential to introduce bias and limit the generalizability and validity of results (Biglan et al., 1987; Fewtrell et al., 2008; Hansen et al., 1990) and is a common occurrence in school-based AOD prevention trials (Ellickson et al., 2003; Vogl et al., 2014). In the present study, comprehensive methods (including emails, phone calls, text messages and letters) were used to attempt to minimize attrition, and follow-up rates of International standards were achieved. In addition, we examined the impact of attrition on our conclusions using multiply imputed data, which is a recommended procedure for handling missing data.

An additional limitation is that some schools required active parental consent which has implications for the generalizability of the results. Although active consent procedures can introduce selection bias (Shaw et al., 2014), other studies found no differences in alcohol or illicit drug use among students with passive and active consent (Anderman et al., 1995). In addition, the majority of students in the present study (64%) received passive consent. If future trials require active consent, attempts should be made to maximize consent rates using recommended strategies (Wolfenden et al., 2009).

A final limitation is the use of self-report survey data. Although self-report might lead to measurement errors, research suggests that it is a reliable and valid method for measuring alcohol (Del Boca and Darkes, 2003) and cannabis use (Ramo et al., 2012). In the current study, assurances of confidentiality were provided and students self-administered the survey online, factors both of which have been shown to enhance accuracy (Brener et al., 2003).

Strengths of the study include the strong follow-up rates and the rigorous design. The present trial not only provides important information about the efficacy of the Climate Schools course but also adds to the prevention science field, in which there is a current dearth of replication. By replicating the program in a new sample of students, we are able to learn more about the strength and generalizability of the intervention. A further strength is that fidelity data indicates that the program was implemented well, primarily due to its Internet-based nature which ensured that core components were retained. This is important given the strong link between high quality implementation and program outcomes (Durlak, 2013) and lends support to the feasibility of using online interventions to deliver AOD prevention. Participant feedback also confirms that teachers and students enjoyed the Climate Schools program, especially the online components. Research suggests that the more favorably teachers rate a program, the more likely they are to implement it well (Han and Weiss, 2005). Although the Climate Schools course is primarily delivered via the Internet, teachers play a key role in fostering discussions after the cartoons, and therefore, it is important that they rate it well.

Conclusion

The present results provide some support for the Internet-based Climate Schools: Alcohol and Cannabis module as a feasible way of delivering alcohol and cannabis prevention. The lack of effects for binge drinking, cannabis use and cannabis intentions highlights the importance of continued follow-up of students to determine longer-term effects of the program. Following this, an important next step will be to identify the underlying factors responsible for the beneficial effects of the Climate Schools: Alcohol and Cannabis course, and what elements can be modified to strengthen program effects, especially in relation to alcohol and cannabis use outcomes.

Footnotes

Acknowledgements

The authors would like to thank the many schools, students, teachers and health professionals who were involved in this research. The research team also acknowledges the assistance of the New South Wales (NSW) Department of Education and Communities for access to its schools (SERAP 2011201).

Declaration of interest

The authors declare that there are no conflicts of interest.

Funding

This study was funded by the National Health and Medical Research Council (APP1004744).

References

Anderman

Cheadle

Curry

. (1995) Selection bias related to parental consent in school-based survey research. Evaluation Review 19: 663–674.

Andrews

Tildesley

Hops

. (2003) Elementary school age children’s future intentions and use of substances. Journal of Clinical Child and Adolescent Psychology 32: 556–567.

Aos

Cook

Elliott

. (2011) Commentary on Valentine, Jeffrey, et al.: Replication in prevention science. The Advisory Board of Blueprints for Violence Prevention. Prevention Science 12: 121–122; discussion 123–125.

Australian Institute of Health and Welfare (AIHW) (2008) 2007 National Drug Strategy Household Survey: Detailed Findings (Drug statistics series no. 22. Cat. no. PHE 107). Canberra, ACT, Australia: AIHW.

Backer

(2001) Finding the Balance: Program Fidelity and Adaptation in Substance Abuse Prevention: A State of the Art Review. Rockville, MD: Center for Substance Abuse Prevention, Substance Abuse and Mental Health Services Administration.

Bennett

Glasgow

(2009) The delivery of public health interventions via the Internet: Actualizing their potential. Annual Review of Public Health 30: 273–292.

Biglan

Steverson

Ary

. (1987) Do smoking prevention programs really work? Attrition and the internal and external validity of an evaluation of refusal skills training program. Journal of Behavioral Medicine 10: 159–171.

Botvin

(2000) Preventing drug abuse in schools: Social and competence enhancement approaches targeting individual-level etiologic factors. Addictive Behaviors 25: 887–897.

Botvin

Griffin

(2007) School-based programmes to prevent alcohol, tobacco and other drug use. International Review of Psychiatry 19: 607–615.

10.

Brener

Billy

JOG

Grady

(2003) Assessment of factors affecting the validity of self-reported health-risk behavior among adolescents: Evidence from the scientific literature. Journal of Adolescent Health 33: 436–457.

11.

Bumbarger

Perkins

(2008) After randomised trials: Issues related to dissemination of evidence-based interventions. Journal of Children’s Services 3: 55–64.

12.

Champion

Newton

Barrett

. (2013) A systematic review of school-based alcohol and other drug prevention programs facilitated by computers or the Internet. Drug and Alcohol Review 32: 115–123.

13.

Chen

Sorr

Anthony

(2009) Early-onset drug use and risk for drug dependence problems. Addictive Behaviors 34: 319–322.

14.

Collins

Lapsley

(2008) The Costs of Tobacco, Alcohol and Illicit Drug Abuse to Australian Society in 2004/05. Canberra, ACT, Australia: Commonwealth of Australia.

15.

Conner

McMillan

(1999) Interaction effects in the theory of planned behaviour: Studying cannabis use. British Journal of Social Psychology 38: 195–222.

16.

Del Boca

Darkes

(2003) The validity of self-reports of alcohol consumption: State of the science and challenges for research. Addiction 98: 1–12.

17.

Durlak

(2013) The Importance of Quality Implementation for Research, Practice, and Policy (ASPE Research Brief). Washington, DC: US Department of Health and Human Services.

18.

Durlak

DuPre

(2008) Implementation matters: A review of research on the influence of implementation on program outcomes and the factors affecting implementation. American Journal of Community Psychology 41: 327–350.

19.

Dusenbury

Brannigan

Hansen

. (2005) Quality of implementation: Developing measures crucial to understanding the diffusion of preventive interventions. Health Education Research 20: 308–313.

20.

Ellickson

McCaffrey

Ghosh-Dastidar

. (2003) New inroads in preventing adolescent drug use: Results from a large-scale trial of project ALERT in middle schools. American Journal of Public Health 93: 1830–1836.

21.

Elliott

Mihalic

(2004) Issues in disseminating and replicating effective prevention programs. Prevention Science 5: 47–53.

22.

Ennett

Ringwalt

Thorne

. (2003) A comparison of current practice in school-based substance use prevention programs with meta-analysis findings. Prevention Science 4: 1–14.

23.

Fewtrell

Kennedy

Singhal

. (2008) How much loss to follow-up is acceptable in long-term randomised trials and prospective studies? Archives of Disease in Childhood 93: 458–461.

24.

Flay

Biglan

Boruch

. (2005) Standards of evidence: Criteria for efficacy, effectiveness and dissemination. Prevention Science 6: 151–175.

25.

Foxcroft

Tsertsvadze

(2011) Universal school-based prevention programs for alcohol misuse in young people (review). Cochrane Database of Systematic Reviews 5: CD009113.

26.

Han

Weiss

(2005) Sustainability of teacher implementation of school-based mental health programs. Journal of Abnormal Child Psychology 33: 665–679.

27.

Hansen

Tobler

Graham

(1990) Attrition in substance abuse prevention research: A meta-analysis of 85 longitudinally followed cohorts. Evaluation Review 14: 677–685.

28.

Henry

(2010) Academic achievement and adolescent drug use: An examination of reciprocal effects and correlated growth trajectories. Journal of School Health 80: 38–43.

29.

Hingson

Heeren

Winter

(2006) Age at drinking onset and alcohol dependence: Age at onset, duration and severity. Archives of Pediatric Adolescent Medicine 160: 739–746.

30.

Kloos

Weller

Chan

. (2009) Gender differences in adolescent substance abuse. Current Psychiatry Reports 11: 120–126.

31.

McBride

Farringdon

Midford

. (2004) Harm minimization in school drug education: Final results of the School Health and Alcohol Harm Reduction Project (SHAHRP). Addiction 99: 278–291.

32.

Manning

Smith

Mazerolle

(2013) The societal costs of alcohol misuse in Australia. In: Trends and Issues in Crime and Criminal Justice. Available at: www.aic.gov.au/media_library/publications/tandi_pdf/tandi454.pdf

33.

Marsch

Bickel

Grabinski

(2007) Application of interactive, computer technology to adolescent substance abuse prevention and treatment. Adolescent Medicine: State of the Art Reviews 18: 342–356, xii.

34.

Midford

Mitchell

Lester

. (2014) Preventing alcohol harm: Early results from a cluster randomised, controlled trial in Victoria, Australia of comprehensive harm minimisation school drug education. International Journal of Drug Policy 25: 142–150.

35.

Mihalic

Fagan

Argamaso

(2008) Implementing the LifeSkills Training drug prevention program: Factors related to implementation fidelity. Implementation Science 3: 5.

36.

National Health and Medical Research Council (2009) Australian Guidelines to Reduce Health Risks from Drinking Alcohol. Canberra, ACT, Australia: Commonwealth of Australia.

37.

Newton

Andrews

Teesson

. (2009) Delivering prevention for alcohol and cannabis using the Internet: A cluster randomised controlled trial. Preventive Medicine 48: 579–584.

38.

Newton

Teesson

Barrett

. (2012) The CAP study, evaluation of integrated universal and selective prevention strategies for youth alcohol misuse: Study protocol of a cluster randomized controlled trial. BMC Psychiatry 12: 118.

39.

Newton

Teesson

Vogl

. (2010) Internet-based prevention for alcohol and cannabis use: Final results of the Climate Schools course. Addiction 105: 749–759.

40.

Pankratz

Jackson-Newsom

Giles

. (2006) Implementation fidelity in a teacher-led alcohol use prevention curriculum. Journal of Drug Education 36: 317–333.

41.

Ramo

Liu

Prochaska

(2012) Reliability and validity of young adults’ anonymous online reports of marijuana use and thoughts about use. Psychology of Addictive Behaviors 26: 801–811.

42.

Royston

(2009) Multiple imputation of missing values: Further update of ice, with an emphasis on categorical variables. Stata Journal 9: 466–477.

43.

Schinke

Schwinn

Di Noia

. (2004) Reducing the risks of alcohol use among urban youth: Three-year effects of a computer-based intervention with and without parent involvement. Journal of Studies on Alcohol 65: 443–449.

44.

Shaw

Cross

Thomas

. (2014) Bias in student survey findings from active parental consent procedures. British Educational Research Journal. Epub ahead of print 24 March. DOI:10.1002/berj.3137.

45.

Sterne

White

Carlin

. (2009) Multiple imputation for missing data in epidemiological and clinical research: Potential and pitfalls. BMJ 338: b2393.

46.

Valentine

Biglan

Boruch

. (2011) Replication in prevention science. Prevention Science 12: 103–117.

47.

Vogl

Newton

Champion

. (2014) A universal harm-minimisation approach to preventing psychostimulant and cannabis use in adolescents: A cluster randomised controlled trial. Substance Abuse Treatment, Prevention, and Policy 9: 24.

48.

Vogl

Teesson

Andrews

. (2009) A computerized harm minimization prevention program for alcohol misuse and related harms: Randomized controlled trial. Addiction 104: 564–575.

49.

White

Bariola

(2012) Australian Secondary School Students’ Use of Tobacco, Alcohol, and Over-the-Counter and Illicit Substances in 2011. Canberra, ACT, Australia: Department of Health and Ageing.

50.

Wolfenden

Kypri

Freund

. (2009) Obtaining active parental consent for school-based research: A guide for researchers. Australian and New Zealand Journal of Public Health 33: 270–275.