Engagement in the Overdose RIsk InfOrmatioN (ORION) e-Health Tool for Opioid Overdose Prevention and Self-Efficacy: A Preliminary Study

Abstract

Increasing awareness of, and information about, overdose risk is an appropriate approach in risk reduction. e-Health technology in substance use disorders is an opportunity to support behavioral changes related to public health concerns. The present study aimed to evaluate the short-term impact of an innovative e-health psychoeducational software, the Overdose RIsk InfOrmatioN (ORION) tool. The ORION programme provided relevant information to opioid-dependent individuals about the risk of suffering a drug overdose as a result of high risky and dysfunctional behaviors. Seven aggregate risk factors were identified through a systematic review and their outputs included in a risk estimation model. We recruited 194 opioid-dependent treatment-seeking individuals from the United Kingdom, Germany, Italy, and Denmark. All participants were given at study entry, and after their use of the software, the General Self-Efficacy (GSE) Scale. We found comparable pre- and post-ORION administration mean GSE scores (SD), 28.49 (5.50) and 28.32 (5.90), respectively (p = 0.297). However, there was an inverse correlation between the number of risk factors and reported levels of self-efficacy (p < 0.001). ORION was able to identify individuals who are most in need of reducing their modifiable risk factors with appropriate interventions. However, a one-shot e-health tool cannot influence complex domains such as self-efficacy unless this is used with other effective interventions. Nonetheless, the ORION tool is unique in its style and content of delivery, that is translating risks combination into a clear estimation, and will need further development such as (a) integration in smartphone-based e-health apps and (b) testing in other high-risk populations.

Introduction

Deaths from opioid overdose have been increasing in the last decades both in the United States¹ and across Europe.² Current effective measures for prevention include provision of appropriate opioid substitution treatment with either methadone or buprenorphine,^3,4 as well as access to take home naloxone.⁵ However, increasing awareness of, and information about, overdose risk factors has potential to support a harm reduction preventative and effective programme to help individuals to change such dysfunctional behaviors.⁶ This is mostly based on the neurocognitive evidence that impaired decision-making, which is actually a common issue for substance users, who show little regard for consequences and often deny or are unaware that they have a problem (e.g., Refs. ^7,8). e-Health technology may encourage behavioral changes related to public health priorities, including those relevant to subjects with substance use disorders.⁹ e-Health tools may enable interventions at a population level in a variety of formats and interventions^10,11 across various substances and for a range of populations and settings.^12,13 The advantages of e-health for people with addiction problems include accessibility and availability, enhanced patient–clinician communication, the provision of information in an engaging manner, the individualization of the intervention, a greater sense of privacy, and reduced stigmatization or embarrassment about drug use.¹⁴ Given the modest beneficial effects of standard overdose prevention interventions,^15,16 e-health tools might obviate some of the difficulties in implementing preventive strategies by taking advantage of users' propensity to use electronic devices.¹⁰

The present study aimed to evaluate the short-term impact of an innovative e-health psychoeducational tool providing information to opioid-dependent individuals about the risk of experiencing a drug overdose. This proof-of-concept study used a pre–posttest design carefully, considering methodological threats such as history, maturation, and regression to the mean.

Materials and Methods

The Overdose RIsk InfOrmatioN tool

Stemming from a European Commission funded project, we aimed at developing an evidence-based risk estimation model running on a PC laptop for flexible utilization in various clinical settings. The tool was designed with the aim of being easy-to-use and informative to both patients and clinicians in various clinical settings, primarily focusing around overdose prevention. We previously reported elsewhere on phase 1 (risk estimation), phase 2 (design), and phase 3 (feasibility) of the Overdose RIsk InfOrmatioN (ORION) study.¹⁷

In brief, aggregate risk factors were identified through a systematic review for inclusion, including for each a specific risk assessment question: injecting behavior (“Do you inject drugs?”), mixing drugs (“Are there days when you take more than one drug [including alcohol]?”), recent (2 weeks) release from prison (“Have you recently been released from prison or residential rehab?”), not receiving treatment (“Are you receiving some form of treatment for taking drugs [including alcohol]?”), using drugs alone (“Have you used drugs [including alcohol] when you were alone?”), no intervention (“Have you tried to reduce your use of drugs [including alcohol]?”), stressful life event (“Have you had an unusually stressful life event, e.g., bereavement, relationship breakup, health problem?”), mental health difficulties (“Are you suffering from a psychological condition, e.g., depression?”), and previous overdoses (“Have you ever been so intoxicated that you were scared of dying?”). Outputs of these factors, included in a risk estimation model, provide a point estimate of relative risk for opioid overdose expressed as a percentage (Fig. 1). A master document compiled in the English language was translated and back translated into German, Italian, and Danish, making available the tool in different languages. The final software, besides general description and legal disclaimer, includes visually engaging and user-friendly screens. ORION tool consists of the following steps: (1) Welcome screen—describing the program and legal disclaimer regarding overdose risk estimation. (2) Demographic information–prompting the users to enter participant number as well as gender and age band. (3) Initial risk assessment questions—nine risk assessment questions with dropdown menus allowing the users to indicate whether or not this particular risk factor applies to them. (4) First overdose risk feedback—displayed by a black marker placed along a horizontal bar ranging from low to high overdose risk (0–100), which was placed against the overdose risk of a nondrug user for comparison. (5) Option to change answers and review of modified risk—participants were given the option to review their answers and visually inspect how different answers are reflected in changes in the overdose risk feedback graphic (Fig. 1). Evaluation questions—three questions about the cognitive, behavioral, and affective impact the software is likely to have on the user. Each had three answer categories: not at all, a little, and very much. (6) Debriefing screen—thanking the participants and explaining that the risk feedback can be recorded and reviewed at a later time. The ORION tool itself recorded the participants responses to the overdose risk questions (screen 3), whether or not participants reviewed their answers to see how this affects risk (screen 4), and which answers in particular were changed (screen 5).¹⁸ Of course several risk factors, for example, recent release from prison was not something with the patient's control. However, the patient in discussion with the responsible clinician worked alongside each other, making clear (e.g., if the patient had been released recently from prison) that he/she needed to improve his/her awareness of risk. This additional contact of a clinical nature was supposed to have the potential (as with the other items) to create a means by which the patient would increase his/her awareness and enable a greater sense of being able to attend positively to these potential risk factors. We expected this increased awareness to improve self-efficacy (see Measures section).

FIG. 1.

Overdose risk questions.

Setting

Recruitment occurred during the same 3-month period in treatment centers across four European countries: United Kingdom, Germany, Italy, and Denmark, in both inpatient and outpatient healthcare settings (NHS Fife Addiction Services in Scotland, Essen LVR-Hospital in Germany; Monza Regional Addiction Service in Italy and Aarhus University Hospital in Denmark, applicants of the relevant European Commission grant awarded). Local ethical governance approval was obtained.

Participants

A pragmatic opportunistic approach was taken, balancing the tight timeframe work packages provided by EU funders and the need to apply through different ethical governance approval processes. We recruited consecutive, male and female, eligible, and consenting patients, aged between 18 and 55 years, who were seeking treatment for their opioid dependence at local clinical services. We excluded individuals with a current history of psychosis, learning disabilities, acute intoxication, and those unable to give informed consent.

Measures

Key workers supported individuals suffering from dependency issues in self-administering the ORION e-health tool on PC laptops, on the first available clinical assessment after informed consent was obtained. All participants were given at study entry, and after their use of the software, the General Self-Efficacy (GSE) Scale.¹⁹ This was chosen based on the Health Action Process Approach (e.g., Ref. ²⁰), assuming that self-efficacy pertains to a sense of control over one's environment and behavior with cognitions that determine whether health behavior change will be initiated, how much effort will be expended, and how long it will be sustained in the face of obstacles and failures. In particular, GSE assesses a broad and stable sense of personal competence to deal effectively with a variety of stressful situations. Individuals may be more or less advanced in the change process (action, maintenance, and recovery), although risk perception, along with outcome expectancies and self-beliefs, plays a major role in this process. In the first phase of the process, the individual does not yet act, but develops a motivation to do so. However, GSE was considered an appropriate measure in this exploratory study to assess impact of ORION on preaction, perceived self-efficacy required to overcome obstacles, to stimulate self-motivation, and to deal with single modifiable risk-factors and overall risk score.

In addition, after ORION tool administration, three brief evaluation questions were asked. These were designed to capture the perceived impact of the software in terms of cognition (new knowledge) “Did you learn anything about drug overdose risk from this program?,” behavior (intention to change) “Is your drug/alcohol taking behavior likely to change as a result of using this information?,” and appraisal (usefulness) “Did you find the program useful?” and were scored on a three-point Likert scale (1 = no; 2 = little; 3 = a lot). These three dimensions are frequently used to capture the impact of healthcare interventions,²¹ distinguishing subgroups of users.

Data analysis

Pre–post-ORION tool differences on GSE were first tested using paired Student's T-test. Single risk factors differences were explored using McNemar's test, testing the difference between paired proportions in our within-subjects design. Pre–post-ORION GSE were also explored by main sociodemographic and clinical characteristics, providing summary statistics (mean and SD). In addition, we investigated differences in post-ORION self-efficacy in terms of perceived impact on the three dimensions considered (cognition, behavior, and appraisal). Furthermore, a linear mixed-effects model was implemented, evaluating pre-/post-ORION administration difference on GSE and the role of ORION risk factors, in terms of relevant number carried by single subjects, controlling for age and gender. Furthermore, since within-country scores might be correlated, with no independent residuals, we adjusted also for country clusters.

Results

We recruited 194 opioid-dependent individuals (United Kingdom, N = 39; Germany, N = 99; Italy, N = 40; and Denmark, N = 16), of whom 155 were receiving outpatient (predominantly opioid maintenance) and 39 inpatient treatment. The average age of participants was 38.8 (SD = 7.9) years, and 86 percent of them were male. Due to missing values in several items on baseline GSE and risk assessment questions, total scores were not available for 23 subjects, who were excluded from the analyses. Furthermore, a small number of subjects (about 10 percent), mainly from the United Kingdom, dropped out during the ORION tool administration. These individuals were generally younger (p = 0.008), with a higher number of overdose risk factors (p = 0.003), and showed, on available items, lower baseline GSE scores, compared with people with complete followup data, mean (SD): 26.5 (3.6) versus 28.5 (5.5), respectively (p = 0.048).

Summary statistics for pre- and post-ORION GSE scores by demographic, risk burden, and impact characteristics are shown in Table 1.

Table 1.

Pre- and Post-Overdose RIsk InfOrmatioN General Self-Efficacy Scores by Demographic, Risk Burden, and Impact Characteristics

	N = 171 (%)	Pre-ORION GSE, mean (SD)	p	N = 154 (%)	Post-ORION GSE, mean (SD)	p
Country			0.461^a			0.725^a
United Kingdom	31 (18)	28.23 (3.89)		16 (10)	29.75 (3.89)
Denmark	7 (4)	27.57 (8.64)		6 (4)	27.33 (9.89)
Germany	93 (54)	28.84 (6.04)		93 (60)	28.32 (6.61)
Italy	40 (23)	27.25 (3.82)		39 (25)	27.87 (3.76)
Gender			0.392^b			0.383^c
Male	146 (85)	28.47 (5.00)		132 (86)	28.56 (5.41)
Female	21 (12)	26.95 (7.76)		19 (12)	26.68 (8.89)
Pre-ORION risk level^d			<0.001^a			<0.001^a
Low (0–2)	56 (33)	30.75 (4.98)		55 (36)	30.75 (5.49)
Medium (3–5)	92 (54)	27.62 (5.05)		81 (53)	27.48 (5.40)
High (>5)	23 (13)	25.09 (5.22)		18 (12)	24.67 (6.60)
Pre-ORION risk factors			—			—
1. Injecting behavior	64 (37)	26.97 (5.12)		54 (35)	26.87 (5.77)
2. Mixing drugs	103 (60)	27.02 (5.11)		89 (58)	26.83 (5.86)
3. Recent release from prison	24 (14)	25.25 (3.91)		17 (11)	25.53 (4.96)
4. Not receiving treatment	17 (10)	28.71 (5.65)		14 (9)	28.50 (6.30)
5. Using drugs alone	117 (68)	27.60 (5.32)		107 (69)	27.28 (5.97)
6. No intervention	47 (27)	29.89 (5.86)		43 (28)	29.88 (6.46)
7. Stressful life event	85 (50)	27.03 (5.42)		71 (46)	26.96 (5.90)
8. Mental health difficulties	86 (50)	26.91 (5.03)		73 (47)	26.78 (5.61)
9. Previous overdoses	39 (23)	25.36 (5.10)		31 (20)	25.26 (6.14)
Perceived impact
New knowledge			0.403^a			0.532^a
I learned nothing	83 (49)	28.86 (5.28)		79 (51)	28.68 (5.75)
I learned a little	73 (43)	27.88 (4.72)		61 (40)	28.20 (5.24)
I learned a lot	15 (9)	27.33 (8.30)		14 (9)	26.79 (8.98)
Intention to change			0.050^a			0.194^a
Not at all	97 (57)	29.11 (5.08)		92 (60)	28.97 (5.55)
Perhaps a little	55 (32)	27.58 (5.26)		46 (30)	27.67 (6.17)
Yes, a lot	19 (11)	26.26 (6.51)		16 (10)	26.44 (6.80)
Appraisal			0.406^a			0.635^a
Not at all useful	31 (18)	28.61 (4.86)		28 (18)	28.68 (5.56)
A little useful	96 (56)	28.64 (5.11)		89 (58)	28.54 (5.45)
Very useful	44 (26)	27.36 (6.22)		37 (24)	27.51 (7.16)

ANOVA.

Student^'s T test with d.f. 22.454.

Student^'s T test with d.f. 19.9594.

No. of risk factors.

d.f., degrees of freedom; ORION, Overdose RIsk InfOrmatioN.

We found comparable pre- and post-ORION administration mean GSE scores (SD), regardless of pre-ORION risk level, distinguished in low (0–2), medium (3–5), and high (>5) as assessed by ORION. Furthermore, with regard to the impact of ORION, about a half of individuals reported that they learnt at least a little about drug overdose risk (51 percent), and perhaps they might change their drug/alcohol taking behavior because of using this information (43 percent). In addition, the majority of individuals considered the ORION programme as useful (little 56 percent and very 26 percent, useful, respectively). However, no univariate statistically significant difference was found in self-efficacy average scores across subgroups by impact on different dimensions (New knowledge, Intention to change, and Appraisal). Exploring single risk factors, we found that those who benefited more from the ORION tool administration were mixing drugs and using drugs alone, both significantly (p < 0.001) reduced as assessed by ORION. Furthermore, evaluating the effect of each of the risk factors, despite longitudinal mean differences, none of these significantly influenced the pre–post-variation in the GSE scores.

A linear mixed-effects model exploring the influence of ORION administration on GSE and the role of the number of reported overdose risk factors, adjusted for country clusters, age and gender, showed no difference (Table 2). The number of overdose risk factors carried by the single individual was confirmed at multivariable analysis as associated with self-efficacy, with a higher number of risk factors resulting in a decreased GSE score.

Table 2.

Linear Mixed-Effects Model for Pre–Post-Overdose RIsk InfOrmatioN General Self-Efficacy

	Regression coefficient (95% CI)	Standard error ^a	p
Time^b	−0.18 (−0.84 to 0.48)	0.33	0.590
Overdose risk factors (N)	−1.21 (−1.42 to −1.00)	0.18	<0.001
Females^c	−2.54 (−3.26 to −1.83)	0.37	<0.001
Age	0.07 (−0.05 to 0.19)	0.06	0.274
Constant	32.67 (28.58 to 36.77)	2.61	<0.001

Standard error adjusted for different country.

pre-ORION.

Males.

Discussion

We explored the beneficial impact of a novel e-health tool for opioid overdose prevention in a sample of subjects in treatment in four different European countries. We had already reported that this tool is user friendly and carries potential advantages.¹⁷ We failed to find any difference in the main outcome measure chosen, that is, self-efficacy, in the overall sample, after individuals used the ORION tool. Nonetheless, participants reported a positive impact of the tool in terms of new knowledge, intention to change, and usefulness. The ORION tool seems thus able to have an impact on more specific outcomes and sensitive measures for risk-communication research.²¹ However, the number of risk factors, as explored by the tool and carried by the single individual, emerged as an important predictor of poor self-efficacy both before and after ORION administration. It could thus be argued that ORION is certainly able to identify subjects who are most in need of reducing their modifiable risk factors with appropriate interventions. However, a one-shot e-health tool cannot influence complex domains such as self-efficacy,²² involving a broad and stable sense of personal competence to deal effectively with a variety of stressful situations. Monitoring modifiable risk factors in routine periodical assessments may be more effective to encourage behavioral change if individual characteristics are taken into account, since these are likely to influence the mediating construct of “readiness to change.”²³ It will therefore be important for the further improvement of the tool to develop comprehensive training to assist potential users (both professional staff and service users). This will highlight the risk assessment with example scenarios and give advisory instructions to assist uptake of the vitally important messages that reside within the engagement of the ORION tool, considering patients bargain with themselves to achieve what they consider an appropriate balance of risk.²⁴ The advantage of the ORION tool may be, with repeated use with a patient, that this balance is recalibrated to achieve an acceptable risk level,²⁵ focusing feasible treatment programs toward modifiable characteristics.¹¹

We acknowledge several limitations. First, our risk estimation model for opioid overdose was built on scientific literature evidence; hence, we lose specificity in terms of local sociodemographic predictors, although generalizability is increased as based on the rigor of selected studies.

In addition, as the ultimate goal of the model is to predict opioid overdose risk from a collection of risk factors, we have to acknowledge that some of these may be unknown for some users. Furthermore, our model shows final point estimates of risk in percentage (e.g., 90 percent “high”). Although this may be useful in decision-making analysis, these numbers by themselves, without confidence intervals, may create a false sense of certainty²⁶ and need to be considered with caution. In addition, we cannot rule out a Hawthorne observer effect, considering that participants knew that their behavior was being tracked. Of course, it would have been a better strategy considering specific elements of self-efficacy, with scales of perceived self-efficacy tailored to the particular domain.²⁷ However, ORION was not able to introduce extensive assessment instruments because it was being applied in a clinical and demanding setting, and in four different countries using different languages. Finally, this proof-of-concept study lacks a control group. This limitation is not unusual in e-health interventions.²⁸ Nonetheless, pre–postdesign possible flaws were sufficiently addressed since it is unlikely that participants can experience different contemporaneous effects of other informative sources (history) or can mature during such a limited amount of time (maturation).²⁹ However, we cannot exclude test effects, attributable to factors such as participants remembering questions or the questions raising awareness and triggering learning after the pretest. Finally, since we did not find any pre–post difference, and we did not select participants on the basis of an extreme pretest score, it is unlikely a regression to the mean phenomenon.³⁰

Despite all these limitations, the ORION e-health tool, available for free download at http://orion-euproject.com/download-software/, underlines the importance of a friendly, evidence-based intervention aimed in obtaining reliable evaluation of overdose risk. It supports a decision-making process aimed at influencing the substance users' self-efficacy and understanding of risk factors. The unique characteristics of the ORION tool, along with its IT- based nature, consist in translating risks combination into a clear estimation. This is likely to make the user more interested in his/her modifiable risk factors. Future work will need to consider the use of this e-health tool also in terms of ecologic validity. This would imply different approaches according to chosen dissemination strategies. Clinicians could actually suggest repeated use of ORION to high-risk opioid dependent subjects, although the integration of this component in standard treatment programs needs to be considered.³¹ Alternatively, viral advertising also using existing social networking services could motivate users, self-selecting to use this e-health tool. This could benefit from integration in other platforms than a laptop device, for example, smartphone-based e-health apps, which would make the tool even more user-friendly and appealing.¹³

Footnotes

Acknowledgments

The authors gratefully acknowledge the support of the following: The users and clinicians from all participating sites. The European Commission, Directorate—General Justice, Freedom and Security. Directorate D: Fundamental Rights and Citizenship: Specific Transnational Projects. DPIP Project No.: Just/2009/Dpip/Ag/0962. The team led by Dr. Simon Thomas at Keele University who developed the software for this project.

Author Disclosure Statement

No competing financial interests exist.

References

CDC—Centers for Disease Control and Prevention, National Center for Health Statistics. (2015) Compressed Mortality File 1999–2014 Series 20 No. 2T on CDC WONDER Online Database, released December 2015. http://wonder.cdc.gov/cmf-icd10.html (accessed Oct. 25, 2016).

EMCDDA—European Monitoring Centre for Drugs and Drug Addiction. (2015) Perspectives on drugs: preventing overdose deaths in Europe. Lisbon: European Monitoring Centre for Drugs and Drug Addiction.

Degenhardt

, Bucello

, Mathers

, et al. Mortality among regular or dependent users of heroin and other opioids: a systematic review and meta-analysis of cohort studies. Addiction, 2011; 106:32–51.

Auriacombe

, Fatséas

, Dubernet

, et al. French field experience with buprenorphine. The American Journal on Addictions, 2004; 13(Suppl. 1):S17–S28.

Strang

, Bird

, Dietze

, et al. Take-home emergency naloxone to prevent deaths from heroin overdose. British Medical Journal, 2014; 349:g6580.

Darke

, Mills

, Ross

, et al. Rates and correlates of mortality amongst heroin users: findings from the Australian Treatment Outcome Study (ATOS), 2001–2009. Drug and Alcohol Dependence, 2011; 115:190–195.

Bechara

. Risky business: emotion, decision-making and addiction. Journal of Gambling Studies, 2003; 19:23–51.

APDES and Akzept. (2014) Preventing avoidable deaths: essentials and recommendations on opioid overdose, EUROHRN, Brussels, 2014. II PROJECT. www.eurohrn.eu/images/stories/pdf/publications/study_report_euroHRN_II.pdf (accessed Oct. 25, 2016).

McClure

, Acquavita

, Harding

, et al. Utilization of communication technology by patients enrolled in substance abuse treatment. Drug and Alcohol Dependence, 2013; 129:145–150.

10.

Kiluk

, Carroll

. New developments in behavioral treatments for substance use disorders. Current Psychiatry Reports, 2013; 15:420.

11.

Carrà

, Crocamo

, Schivalocchi

, et al. Risk estimation modeling and feasibility testing for a mobile ehealth intervention for binge drinking among young people: the D-ARIANNA (Digital-Alcohol RIsk Alertness Notifying Network for Adolescents and young adults) Project. Substance Abuse, 2015; 36:445–452.

12.

Marsch

, Carroll

, Kiluk

. Technology-based interventions for the treatment and recovery management of substance use disorders: a JSAT special issue. Journal of Substance Abuse Treatment, 2014; 46:1–4.

13.

Carrà

, Crocamo

, Bartoli

, et al. Impact of a mobile e-Health intervention on binge drinking in young people: the digital-alcohol risk alertness notifying network for adolescents and young adults project. The Journal of Adolescent Health, 2016; 58:520–526.

14.

Wood

, Eckley

, Hughes

, et al. Computer-based programmes for the prevention and management of illicit recreational drug use: a systematic review. Addictive Behaviors, 2014; 39:30–38.

15.

McCambridge

, Strang

. The efficacy of single-session motivational interviewing in reducing drug consumption and perceptions of drug-related risk and harm among young people: results from a multi-site cluster randomized trial. Addiction, 2004; 99:39–52.

16.

Clark

, Wilder

, Winstanley

. A systematic review of community opioid overdose prevention and naloxone distribution programs. Journal of Addiction Medicine, 2014; 8:153–163.

17.

Baldacchino

, Crocamo

, Humphris

, et al. Decision support in addiction: the development of an e-health tool to assess and prevent risk of fatal overdose. The ORION Project. Computer Methods and Programs in Biomedicine, 2016; 133:207–216.

18.

Humphris

, Carrà

, Frisher

, et al. Engagement in an E-health tool (ORION) predicts opioid-dependent patient likelihood of behavioural change. Heroin Addiction and Related Clinical Problems, 2017; 5:35–44.

19.

Schwarzer

, Jerusalem

. (1995) Generalized self-efficacy scale. In Weinman

, Wright

, Johnston

, eds. Measures in health psychology: a user's portfolio. Causal and control beliefs. Windsor, UK: NFER-NELSON, pp. 35–37.

20.

Schwarzer

, Renner

. Social-cognitive predictors of health behavior: action self-efficacy and coping self-efficacy. Health Psychology, 2000; 19:487–495.

21.

Edwards

, Elwyn

. How should effectiveness of risk communication to aid patients' decisions be judged? A review of the literature. Medical Decision Making, 1999; 19:428–434.

22.

Finney Rutten

, Hesse

, St Sauver

, et al. Health self-efficacy among populations with multiple chronic conditions: the value of patient-centered communication. Advances in Therapy, 2016; 33:1440–1451.

23.

Wolever

, Webber

, Meunier

, et al. Modifiable disease risk, readiness to change, and psychosocial functioning improve with integrative medicine immersion model. Alternative Therapies in Health and Medicine, 2011; 17:38–47.

24.

Rowe

, Santos

, Behar

, et al. Correlates of overdose risk perception among illicit opioid users. Drug and Alcohol Dependence, 2016; 159:234–239.

25.

Rabiau

, Knauper

, Miquelon

. The eternal quest for optimal balance between maximizing pleasure and minimizing harm: the compensatory health beliefs model. British Journal of Health Psychology, 2006; 11:139–153.

26.

Schwartz

, Woloshin

, Welch

. Risk communication in clinical practice: putting cancer in context. Journal of the National Cancer Institute. Monographs, 1999; 25:124–133.

27.

Bandura

. (2006) Guide for constructing self-efficacy scales. In Pajares

, Urdan

, eds. Self-efficacy beliefs of adolescents. Greenwich, CT: Information Age Publishing, vol. 5, pp. 307–337.

28.

Harris

, Mcgregor

, Perencevich

, et al. The use and interpretation of quasi-experimental studies in medical informatics. Journal of the American Medical Informatics Association, 2006; 13:16–23.

29.

Marsden

, Torgerson

. Single group pre- and post-test designs: some methodological concerns. Oxford Review of Education, 2012; 38: 583–616.

30.

Cook

, Campbell

. (1979) Quasi-experimentation: design and analysis issues for field settings. Boston, MA: Houghton Mifflin, vol. 351.

31.

Semisa

, Casacchia

, Di Munzio

, et al. [Promoting recovery of schizophrenic patients: discrepancy between routine practice and evidence. The SIEP-DIRECT'S Project]. Epidemiologia e Psichiatria Sociale, 2008; 17:331–348.