Survey Design Elements as Influences on Estimates of Self-Reported Illicit Substance Use and Other Illegal Activities

Abstract

National surveys of self-reported substance use and other problem behavior frequently produce different prevalence estimates for substance use and other problem behaviors. There is research indicating different aspects of survey methodology may artificially inflate or deflate prevalence rates, and while some research has been done on specific influences, taken separately, little has been done to estimate the cumulative impact of these methodological variations, taken together, on survey estimates. The present study compares methodologies of three studies, Monitoring the Future (MTF), National Survey on Drug Use and Health (NSDUH), and National Youth Survey Family Study (NYSFS), on how differences in methodology may influence estimated rates of problem behavior, and the extent to which knowledge of the differences, taken in combination, allow prediction of differences in those rates. Differences in survey methodology are predictive of differences in prevalence. Implications for the use of the surveys are considered.

Keywords

substance use illegal behaviors adolescents self-report data survey methods

Get full access to this article

View all access options for this article.

References

Anderman

Cheadle

Curry

Diehr

Shultz

Wagner

(1995). Selection bias related to parental consent in school-based survey research. Evaluation Review, 19, 663-674.

Bachman

J. G.

Johnston

L. D.

O’Malley

P. M.

(2001). The Monitoring the Future project after twenty-seven years: Design and procedures (MTF Occasional Paper, 54). Ann Arbor, MI: Institute of Social Research.

Brener

N. D.

Grunbaum

J. A.

Kann

McManus

Ross

(2004). Assessing health risk behaviors among adolescents: The effect of question wording and appeals for honesty. Journal of Adolescent Health, 35, 91-100.

Cantor

(1989). Substantive implications of longitudinal design features: The National Crime Survey as a case study. In Kasprzyk

Duncan

Kalton

Singh

M. P.

(Eds.), Panel surveys (pp. 35-51). New York, NY: Wiley.

Dillman

D. A.

Singer

Clark

J. R.

Treat

J. B.

(1996). Effects of benefits appeals, mandatory appeals, and variations in statements on confidentiality on completion rates for census questionnaires. Public Opinion Quarterly, 60, 376-389.

Doleschal

(1970). Review: Hidden crime. Crime and Delinquency Literature, 2, 546-572.

Elliott

D. S.

Huizinga

Menard

(1989). Multiple problem youth: Delinquency, substance use, and mental health problems. New York, NY: Springer-Verlag.

Esbensen

Deschenes

E. P.

Vogel

R. E.

West

Arboit

Harris

(1996). Active parental consent in school-based research: An examination of ethical and methodological issues. Evaluation Review, 20, 737-753.

Farrington

D. P.

(1973). Self-reports of deviant behavior: Predictive and stable? Journal of Criminal Law and Criminology, 64, 99-110.

10.

Fendrich

Johnson

T. P.

(2001). Examining prevalence differences in three national surveys of youth: Impact of consent procedures, mode, and editing rules. Journal of Drug Issues, 31, 615-643.

11.

Fowler

F. J.

Jr. Stringfellow

V. L.

(2001). Learning from experience: Estimating teen use of alcohol, cigarettes, and marijuana from three survey protocols. Journal of Drug Issues, 31, 643-664.

12.

Fox

J. A.

Tracy

P. E.

(1986). Randomized response: A method for sensitive surveys. Beverly Hills, CA: Sage.

13.

Gfroerer

Wright

Kopstein

(1997). Prevalence of youth substance use: The impact of methodological differences between two national surveys. Drug and Alcohol Dependence, 47, 19-30.

14.

Gold

(1966). Undetected delinquent behavior. Journal of Research in Crime & Delinquency, 3, 27-46.

15.

Gold

(1970). Delinquent behavior in an American city. Belmont, CA: Brooks/Cole.

16.

Hardt

R. H.

Bodine

G. E.

(1965). Development of self-report instruments in delinquency research: A conference report. Syracuse, NY: Syracuse University Youth Development Center.

17.

Harrison

L. D.

(2001). Understanding the differences in youth drug prevalence rates produced by the MTF, NHSDA, and YRBS studies. Journal of Drug Issues, 31, 665-694.

18.

Hennessy

K. H.

Ginsberg

(Eds.). (2001). Substance use survey data collection methodologies: Introduction. Journal of Drug Issues, 31, 595-598.

19.

Hindelang

M. J.

Hirschi

Weis

J. G.

(1979). Correlates of delinquency: The illusion of discrepancy between self-report and official measures. American Sociological Review, 44, 995-1014.

20.

Johnston

L. D.

Bachman

J. G.

O’Malley

P. M.

Schulenberg

J. E.

(2003a). Monitoring the future: A continuing Study of American Youth (8th- and 10th-Grade Surveys), 2003 [Computer file]. Conducted by University of Michigan, Survey Research Center. ICPSR04018-v2. Ann Arbor, MI: Inter-university Consortium for Political and Social Research [producer and distributor], 2004-10-29.

21.

Johnston

L. D.

Bachman

J. G.

O’Malley

P. M.

Schulenberg

J. E.

(2003b). Monitoring the future: A continuing Study of American Youth (12th-Grade Survey), 2003 [Computer file]. Conducted by University of Michigan, Survey Research Center. ICPSR02751-v1. Ann Arbor, MI: Inter-university Consortium for Political and Social Research [producer and distributor], 2009-10-29.

22.

Johnston

L. D.

O’Malley

P. M.

Bachman

J. G.

Schulenberg

J. E.

(2004). Demographic subgroup trends for various licit and illicit drugs 1975-2003 (MTF Occasional Paper No., 60). Ann Arbor, MI: Institute for Social Research.

23.

Kearney

K. A.

Hopkins

R. H.

Mauss

A. L.

Weisheit

R. A.

(1983). Sample bias resulting from a requirement for written parental consent. Public Opinion Quarterly, 47, 96-102.

24.

Kulik

J. A.

Stein

K. B.

Sarbin

T. R.

(1968). Disclosure of delinquent behavior under conditions of anonymity and non-anonymity. Journal of Consulting and Clinical Psychology, 32, 506-509.

25.

Lehnen

R. G.

Skogan

W. G.

(1981). The National Crime Survey: Working papers, Volume 1—Current and historical perspectives. Washington, DC: U.S. Department of Justice.

26.

Martinez-Ebers

(1997). Using monetary incentives with hard-to-reach populations in panel surveys. International Journal of Public Opinion Research, 9, 77-86.

27.

Menard

(Ed.). (2002). Longitudinal research. Thousand Oaks, CA: Sage.

28.

Menard

(2012). Age, criminal victimization, and offending: Changing relationships from adolescence to middle adulthood. Victims & Offenders, 7, 227-254.

29.

Menard

Elliott

D. S.

(1993). Data set comparability and short-term trends in crime and delinquency. Journal of Criminal Justice, 21, 433-445.

30.

Menard

Morris

R. G.

Gerber

Covey

H. C.

(2011). Distribution and correlates of self-reported crimes of trust. Deviant Behavior, 32, 877-917.

31.

O’Malley

P. M.

Johnston

L. D.

Bachman

J. B.

Schulenberg

(2000). A comparison of confidential versus anonymous survey procedures: Effects on reporting of drug issues and related attitudes and beliefs in a national study of students. Journal of Drug Issues, 30(10), 35-44.

32.

Singer

Bossarte

R. M.

(2006). Incentives for survey participation: When are they “coercive.” American Journal of Preventive Medicine, 31, 411-418.

33.

Singer

Von Thurn

D. R.

Miller

E. R.

(1995). Informed consent: Consequences for response rate and response quality in social surveys. American Sociological Review, 43, 144-162.

34.

Substance Abuse and Mental Health Services Administration. (2003). Results from the 2002 National Survey on Drug Use and Health: National findings (Office of Applied Studies NHSDA Series H–22, DHHS Pub. No. SMA 03–3836.). Rockville, MD: U.S. Department of Health and Human Services.

35.

Substance Abuse and Mental Health Services Administration. (2006). Results from the 2005 National Survey on Drug Use and Health: National findings (Office of Applied Studies, NSDUH Series H-30, DHHS Publication No. SMA 06–4194). Rockville, MD: U.S. Department of Health and Human Services.

36.

Sudman

(2001). Examining substance abuse data collection methodologies. Journal of Drug Issues, 31, 695-716.

37.

Sudman

Bradburn

N. M.

(1974). Response effects in surveys. Chicago, IL: Aldine.

38.

Unger

J. B.

Gallaher

Palmer

P. H.

Baezconde-Garbanati

Trinidad

D. R.

Cen

Johnson

C. A.

(2004). No news is bad news: Characteristics of adolescents who provide neither parental consent nor refusal for participation in survey research. Evaluation Review, 28, 52-63.

39.

United States Department of Health and Human Services, Substance Abuse and Mental Health Services Administration, Office of Applied Studies. (2006). National Survey on Drug Use and Health, 2003 (ICPSR 4138) [Computer file]. ICPSR version. Research Triangle Park, NC: Research Triangle Institute [producer], 2004. Ann Arbor, MI: Inter-University Consortium for Political and Social Research [distributor], 2004.

40.

Wells

L. E.

Rankin

J. H.

(1995). Juvenile victimization: Convergent validation of alternative measurements. Journal of Research in Crime and Delinquency, 32(3), 287-307.