Estimation of a Relative Risk Effect Size when Using Continuous Outcomes Data: An Application of Methods in the Prevention of Major Depression and Eating Disorders
Restricted accessResearch articleFirst published online October, 2018
Estimation of a Relative Risk Effect Size when Using Continuous Outcomes Data: An Application of Methods in the Prevention of Major Depression and Eating Disorders
Introduction. The raw mean difference (RMD) and standardized mean difference (SMD) are continuous effect size measures that are not readily usable in decision-analytic models of health care interventions. This study compared the predictive performance of 3 methods by which continuous outcomes data collected using psychiatric rating scales can be used to calculate a relative risk (RR) effect size. Methods. Three methods to calculate RR effect sizes from continuous outcomes data are described: the RMD, SMD, and Cochrane conversion methods. Each conversion method was validated using data from randomized controlled trials (RCTs) examining the efficacy of interventions for the prevention of depression in youth (aged ≤17 years) and adults (aged ≥18 years) and the prevention of eating disorders in young women (aged ≤21 years). Validation analyses compared predicted RR effect sizes to actual RR effect sizes using scatterplots, correlation coefficients (r), and simple linear regression. An applied analysis was also conducted to examine the impact of using each conversion method in a cost-effectiveness model. Results. The predictive performances of the RMD and Cochrane conversion methods were strong relative to the SMD conversion method when analyzing RCTs involving depression in adults (RMD: r = 0.89–0.90; Cochrane: r = 0.73; SMD: r = 0.41–0.67) and eating disorders in young women (RMD: r = 0.89; Cochrane: r = 0.96). Moderate predictive performances were observed across the 3 methods when analyzing RCTs involving depression in youth (RMD: r = 0.50; Cochrane: r = 0.47; SMD: r = 0.46–0.46). Negligible differences were observed between the 3 methods when applied to a cost-effectiveness model. Conclusion. The RMD and Cochrane conversion methods are both valid methods for predicting RR effect sizes from continuous outcomes data. However, further validation and refinement are required before being applied more broadly.
EatonWWHallALMacdonaldRMcKibbenJ.Case identification in psychiatric epidemiology: a review. Int Rev Psychiatry. 2007;19(5):497–507.
2.
StuartALPascoJAJackaFNBrennanSLBerkMWilliamsLJ.Comparison of self-report and structured clinical interview in the identification of depression. Compr Psychiatry. 2014;55(4):866–9.
3.
BaerLBlaisMA, eds. Handbook of Clinical Rating Scales and Assessment in Psychiatry and Mental Health. Totowa, NJ: Humana Press; 2010.
4.
SajatovicMRamirezLF.Rating Scales in Mental Health. 3rd ed.Baltimore, MA: Johns Hopkins University Press; 2012.
5.
BeckATWardCHMendelsonMMockJErbaughJ.An inventory for measuring depression. Arch Gen Psychiatry. 1961;4:561–71.
6.
BeckATSteerRABrownGK.Manual for the Beck Depression Inventory–II. San Antonio, TX: Psychological Corporation; 1996.
7.
SticeETelchCFRizviSL.Development and validation of the Eating Disorder Diagnostic Scale: a brief self-report measure of anorexia, bulimia, and binge-eating disorder. Psychol Assess. 2000;12(2):123.
8.
StockingsEDegenhardtLLeeYYet al. Symptom screening scales for detecting major depressive disorder in children and adolescents: a systematic review and meta-analysis of reliability, validity and diagnostic utility. J Affect Disord. 2015;174:447–63.
9.
KrabbenborgMAMDannerUNLarsenJKet al. The Eating Disorder Diagnostic Scale: psychometric features within a clinical population and a cut-off point to differentiate clinical patients from healthy controls. Eur Eat Disord Rev. 2012;20(4):315–20.
10.
HigginsJPTGreenS.Cochrane Handbook for Systematic Reviews of Interventions. New York, NY: John Wiley; 2011.
11.
BaguleyT.Standardized or simple effect size: what should be reported?Br J Psychol. 2009;100(Pt 3):603–17.
12.
LeLK-DBarendregtJHayPMihalopoulosC. Prevention of eating disorders: a systematic review and meta-analysis. Clin Psychol Rev. 2017;53:46–58.
13.
StockingsEADegenhardtLDobbinsTet al. Preventing depression and anxiety in young people: a review of the joint efficacy of universal, selective and indicated prevention. Psychol Med. 2016;46(1):11–26.
14.
BorensteinMHedgesLVHigginsJPTRothsteinHR.Introduction to Meta-analysis. Chichester, UK: John Wiley; 2009.
15.
McGrathREMeyerGJ.When effect sizes disagree: the case of r and d. Psychol Methods. 2006;11(4):386–401.
16.
CohenJ.Statistical Power Analysis for the Behavioral Sciences. 2nd ed.Hillsdale, NJ: Lawrence Erlbaum; 1988.
17.
BriggsAHClaxtonKSculpherMJ.Decision Modelling for Health Economic Evaluation. Oxford, UK: Oxford University Press; 2006.
18.
Tan-Torres EdejerTBaltussenRAdamTet al. Making Choices in Health: WHO Guide to Cost-Effectiveness Analysis. Geneva, Switzerland: World Health Organization; 2003.
19.
WeinsteinMCO’BrienBHornbergerJet al. Principles of good practice for decision analytic modeling in health-care evaluation: report of the ISPOR Task Force on Good Research Practices—Modeling Studies. Value Health. 2003;6(1):9–17.
20.
BarendregtJJ.The effect size in uncertainty analysis. Value Health. 2010;13(4):388–91.
21.
ChinnS.A simple method for converting an odds ratio to effect size for use in meta-analysis. Stat Med. 2000;19(22):3127–31.
22.
HasselbladVHedgesLV.Meta-analysis of screening and diagnostic tests. Psychol Bull. 1995;117(1):167–78.
23.
MurrayCJLEzzatiMLopezADRodgersAVander HoornS. Comparative quantification of health risks conceptual framework and methodological issues. Popul Health Metr. 2003;1(1):1.
24.
EzzatiMLopezADRodgersAMurrayCJL, eds. Comparative Quantification of Health Risks: Global and Regional Burden of Disease Attributable to Selected Major Risk Factors. Geneva, Switzerland: World Health Organization; 2004.
25.
BriggsAHWeinsteinMCFenwickEAet al. Model parameter estimation and uncertainty: a report of the ISPOR-SMDM Modeling Good Research Practices Task Force–6. Value Health. 2012;15(6):835–42.
26.
AltmanDMachinDBryantTGardnerM, eds. Statistics with Confidence: Confidence Intervals and Statistical Guidelines. 2nd ed.Bristol, UK: BMJ Books; 2000.
27.
BarendregtJJVeermanJL.Categorical versus continuous risk factors and the calculation of potential impact fractions. J Epidemiol Community Health. 2010;64(3):209–12.
28.
DoiSARBarendregtJJKhanSThalibLWilliamsGM.Advances in the meta-analysis of heterogeneous clinical trials I: the inverse variance heterogeneity model. Contemp Clin Trials. 2015;45(Pt A):130–8.
29.
CuijpersPvan StratenASmitFMihalopoulosCBeekmanA.Preventing the onset of depressive disorders: a meta-analytic review of psychological interventions. Am J Psychiatry. 2008;165(10):1272–80.
30.
MerrySMcDowellHHetrickSBirJMullerN.Psychological and/or educational interventions for the prevention of depression in children and adolescents. Cochrane Database Syst Rev. 2004;(1):CD003380.
31.
van ZoonenKBuntrockCEbertDDet al. Preventing the onset of major depressive disorder: a meta-analytic review of psychological interventions. Int J Epidemiol. 2014;43(2):318–29.
32.
WatsonHJJoyceTFrenchEet al. Prevention of eating disorders: a systematic review of randomized, controlled trials. Int J Eat Disord. 2016;49(9):833–62.
33.
PiñeiroGPerelmanSGuerschmanJPParueloJM.How to evaluate models: observed vs. predicted or predicted vs. observed?Ecol Modell. 2008;216(3–4):316–22.
34.
CampbellMJSwinscowTDV. Statistics at Square One. Chichester, UK: John Wiley; 2009.
35.
R Core Team. R: A Language and Environment for Statistical Computing. Vienna, Austria: R Foundation for Statistical Computing; 2017. Available from: https://www.R-project.org/
36.
UherRFarmerAMaierWet al. Measuring depression: comparison and integration of three scales in the GENDEP study. Psychol Med. 2008;38(2):289–300.
37.
WahlILoweBBjornerJBet al. Standardization of depression measurement: a common metric was developed for 11 self-report depression measures. J Clin Epidemiol. 2014;67(1):73–86.
38.
LeeYYBarendregtJJStockingsEAet al. The population cost-effectiveness of delivering universal and indicated school-based interventions to prevent the onset of major depression among youth in Australia. Epidemiol Psychiatr Sci. 2017;26(5):545–64.
39.
CarterRVosTMoodieMHabyMMagnusAMihalopoulosC.Priority setting in health: origins, description and application of the Australian Assessing Cost-Effectiveness initiative. Expert Rev Pharmacoecon Outcomes Res. 2008;8(6):593–617.
40.
VosTHabyMMMagnusAMihalopoulosCAndrewsGCarterR.Assessing cost-effectiveness in mental health: helping policy-makers prioritize and plan health services. Aust N Z J Psychiatry. 2005;39(8):701–12.
41.
Allart-van DamEHosmanCMHHoogduinCALSchaapCPDR.The Coping with Depression course: short-term outcomes and mediating effects of a randomized controlled trial in the treatment of subclinical depression. Behav Ther. 2003;34(3):381–96.
42.
BohlmeijerETFledderusMRokxTAPieterseME.Efficacy of an early intervention based on acceptance and commitment therapy for adults with depressive symptomatology: evaluation in a randomized controlled trial. Behav Res Ther. 2011;49(1):62–7.
43.
FledderusMBohlmeijerETPieterseMESchreursKM.Acceptance and commitment therapy as guided self-help for psychological distress and positive mental health: a randomized controlled trial. Psychol Med. 2012;42(3):485–95.
44.
GeisnerIMNeighborsCLarimerME.A randomized clinical trial of a brief, mailed intervention for symptoms of depression. J Consult Clin Psychol. 2006;74(2):393–9.
45.
PaceTMDixonDN.Changes in depressive self-schemata and depressive symptoms following cognitive therapy. J Couns Psychol. 1993;40(3):288–94.
46.
SeligmanMESchulmanPTryonAM.Group prevention of depression and anxiety symptoms. Behav Res Ther. 2007;45(6):1111–26.
47.
PedenARHallLARayensMKBeebeLL.Reducing negative thinking and depressive symptoms in college women. J Nurs Scholarsh. 2000;32(2):145–51.
48.
Perez-StableEJMirandaJMunozRFYingYW.Depression in medical outpatients: underrecognition and misdiagnosis. Arch Intern Med. 1990;150(5):1083–8.
49.
DuttonGRGrotheKBJonesGNWhiteheadDKendraKBrantleyPJ.Use of the Beck Depression Inventory–II with African American primary care patients. Gen Hosp Psychiatry. 2004;26(6):437–42.
50.
Gomes-OliveiraMHGorensteinCLotufo NetoFAndradeLHWangYP.Validation of the Brazilian Portuguese version of the Beck Depression Inventory–II in a community sample. Rev Bras Psiquiatr. 2012;34(4):389–94.
51.
KjaergaardMArfwedson WangCEWaterlooKJordeR.A study of the psychometric properties of the Beck Depression Inventory–II, the Montgomery and Asberg Depression Rating Scale, and the Hospital Anxiety and Depression Scale in a sample from a healthy population. Scand J Psychol. 2014;55(1):83–9.
52.
Campo-AriasADiaz-MartinezLARueda-JaimesGEdel Pilar CadenaLHernandezNL.Psychometric properties of the CES-D Scale among Colombian adults from the general population. Rev Colomb Psiquiatr. 2007;36(4):664–74.
53.
LiZHicksMH.The CES-D in Chinese American women: construct validity, diagnostic validity for major depression, and cultural response bias. Psychiatry Res. 2010;175(3):227–32.
54.
RingJMMarquisP.Depression in a Latino immigrant medical population: an exploratory screening and diagnosis. Am J Orthopsychiatry. 1991;61(2):298–302.
55.
ThomasJLJonesGNScarinciICMehanDJBrantleyPJ.The utility of the CES-D as a depression screening measure among low-income women attending primary care clinics. The Center for Epidemiologic Studies–Depression. Int J Psychiatry Med. 2001;31(1):25–40.
56.
WadaKTanakaKTheriaultGet al. Validity of the Center for Epidemiologic Studies Depression Scale as a screening instrument of major depressive disorder among Japanese workers. Am J Ind Med. 2007;50(1):8–12.
57.
CanalsJBladéJCarbajoGDomènech-LLaberíaE.The Beck Depression Inventory: psychometric characteristics and usefulness in nonclinical adolescents. Eur J Psychol Assess. 2001;17(1):63–8.
58.
KashaniJHShermanDDParkerDRReidJC.Utility of the Beck Depression Inventory with clinic-referred adolescents. J Am Acad Child Adolesc Psychiatry. 1990;29(2):278–82.
59.
RobertsRELewinsohnPMSeeleyJR.Screening for adolescent depression: a comparison of depression scales. J Am Acad Child Adolesc Psychiatry. 1991;30(1):58–66.
60.
RussellPSBaskerMRussellSMosesPDNairMKMinjuKA.Comparison of a self-rated and a clinician-rated measure for identifying depression among adolescents in a primary-care setting. Indian J Pediatr. 2012;79(Suppl 1):S45–51.
61.
CuijpersPBoluijtPvan StratenA.Screening of depression in adolescents through the Internet: sensitivity and specificity of two screening questionnaires. Eur Child Adolesc Psychiatry. 2008;17(1):32–8.
62.
PrescottCAMcArdleJJHishinumaESet al. Prediction of major depression and dysthymia from CES-D scores among ethnic minority adolescents. J Am Acad Child Adolesc Psychiatry. 1998;37(5):495–503.
63.
ChristensenHBatterhamPJGoslingJAet al. Effectiveness of an online insomnia program (SHUTi) for prevention of depressive episodes (the GoodNight Study): a randomised controlled trial. Lancet Psychiatry. 2016;3(4):333–41.
64.
OteroPSmitFCuijpersPTorresABlancoVVazquezFL.Long-term efficacy of indicated prevention of depression in non-professional caregivers: randomized controlled trial. Psychol Med. 2015;45(7):1401–12.
65.
SeligmanMEPSchulmanPDeRubeisRJHollonSD. The prevention of depression and anxiety. Prev Treat. 1999;2(1):8a.
66.
WillemseGRSmitFCuijpersPTiemensBG.Minimal-contact psychotherapy for sub-threshold depression in primary care: randomised trial. Br J Psychiatry. 2004;185:416–21.
67.
ZhangDXLewisGArayaRet al. Prevention of anxiety and depression in Chinese: a randomized clinical trial testing the effectiveness of a stepped care program in primary care. J Affect Disord. 2014;169:212–20.
68.
ClarkeGNHawkinsWMurphyMSheeberLBLewinsohnPMSeeleyJR.Targeted prevention of unipolar depressive disorder in an at-risk sample of high school adolescents: a randomized trial of a group cognitive intervention. J Am Acad Child Adolesc Psychiatry. 1995;34(3):312–21.
69.
SticeERohdePSeeleyJRGauJM.Brief cognitive-behavioral depression prevention program for high-risk adolescents outperforms two alternative interventions: a randomized efficacy trial. J Consult Clin Psychol. 2008;76(4):595–606.
70.
YoungJFMufsonLDaviesM.Efficacy of interpersonal psychotherapy-adolescent skills training: an indicated preventive intervention for depression. J Child Psychol Psychiatry. 2006;47(12):1254–62.
71.
SticeERohdePButrynMLShawHMartiCN.Effectiveness trial of a selective dissonance-based eating disorder prevention program with female college students: effects at 2- and 3-year follow-up. Behav Res Ther. 2015;71:20–6.
72.
SticeERohdePShawHGauJ.An effectiveness trial of a selected dissonance-based eating disorder prevention program for female high school students: long-term effects. J Consult Clin Psychol. 2011;79(4):500–8.
73.
SticeERohdePShawHMartiCN.Efficacy trial of a selective prevention program targeting both eating disorders and obesity among female college students: 1- and 2-year follow-up effects. J Consult Clin Psychol. 2013;81(1):183–9.
74.
GillhamJEReivichKJBrunwasserSMet al. Evaluation of a group cognitive-behavioral depression prevention program for young adolescents: a randomized effectiveness trial. J Clin Child Adolesc Psychol. 2012;41(5):621–39.
75.
LockSBarrettPM.A longitudinal study of developmental differences in universal preventive intervention for child anxiety. Behav Change. 2003;20(4):183–99.
76.
Lowry-WebsterHMBarrettPMDaddsMR.A universal prevention trial of anxiety and depressive symptomatology in childhood: preliminary data from an Australian study. Behav Change. 2012;18(1):36–50.
77.
ManassisKWilansky-TraynorPFarzanNKleimanVParkerKSanfordM.The feelings club: randomized controlled evaluation of school-based CBT for anxious or depressive symptoms. Depress Anxiety. 2010;27(10):945–52.
78.
RohdePSticeEShawHBriereFN.Indicated cognitive behavioral group depression prevention compared to bibliotherapy and brochure control: acute effects of an effectiveness trial with adolescents. J Consult Clin Psychol. 2014;82(1):65–74.
79.
RooneyRHassanSKaneRRobertsCMNesaM.Reducing depression in 9-10 year old children in low SES schools: a longitudinal universal randomized controlled trial. Behav Res Ther. 2013;51(12):845–54.
80.
RooneyRRobertsCKaneRet al. The prevention of depression in 8- to 9-year-old children: a pilot study. Aust J Guid Counsell. 2006;16(1):76–90.
81.
SheffieldJKSpenceSHRapeeRMet al. Evaluation of universal, indicated, and combined cognitive-behavioral approaches to the prevention of depression among adolescents. J Consult Clin Psychol. 2006;74(1):66–79.
82.
CholletFTardyJAlbucherJFet al. Fluoxetine for motor recovery after acute ischaemic stroke (FLAME): a randomised placebo-controlled trial. Lancet Neurol. 2011;10(2):123–30.
83.
CompasBEForehandRKellerGet al. Randomized controlled trial of a family cognitive-behavioral preventive intervention for children of depressed parents. J Consult Clin Psychol. 2009;77(6):1007–20.
84.
GarberJClarkeGNWeersingVRet al. Prevention of depression in at-risk adolescents: a randomized controlled trial. JAMA. 2009;301(21):2215–24.
85.
GillhamJEHamiltonJFreresDRPattonKGallopR.Preventing depression among early adolescents in the primary care setting: a randomized controlled study of the Penn Resiliency Program. J Abnorm Child Psychol. 2006;34(2):203–19.
86.
MassonSCTejaniAM.Minimum clinically important differences identified for commonly used depression rating scales. J Clin Epidemiol. 2013;66(7):805–7.
87.
BeckATSteerRAGarbinMG.Psychometric properties of the Beck Depression Inventory: twenty-five years of evaluation. Clin Psychol Rev. 1988;8(1):77–100.
88.
RadloffLS.The CES-D scale: a self-report depression scale for research in the general population. Appl Psychol Meas. 1977;1(3):385–401.
89.
SticeEMartiCNSpoorSPresnellKShawH.Dissonance and healthy weight eating disorder prevention programs: long-term effects from a randomized efficacy trial. J Consult Clin Psychol. 2008;76(2):329–40.
90.
SticeEShawH.Prospective relations of body image, eating, and affective disturbances to smoking onset in adolescent girls: how Virginia slims. J Consult Clin Psychol. 2003;71(1):129–35.
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
