The Defense Manpower Data Center of the U.S. Department of Defense launched the 2006 Survey of Active Duty Spouses to assess attitudes of the spouses of U.S. active-duty military members. Severe problems existed with the sampling, weight adjustments, and estimation (including variance-estimation) procedures. Stratification of the sample without proper consideration of the survey objectives made it impossible to achieve reportable information for many desired population subgroups. Excessive stratification caused many of the sampling strata to have very small numbers of respondents, both expected and actual. Consequently, nonresponse bias was probably enormous across many of the strata. Absurd weight adjustments likely contributed toward rendering many survey estimates unreliable. To make the survey estimates seem more precise, sampling strata were collapsed together to form new “variance strata” for variance estimation. Caution is advised in using results from this and other Defense Manpower Data Center surveys.
Defense Manpower Data Center, "1999 Survey of Spouses of Active Duty Members: Statistical Methodology Report," report no. 2000-021 ( Arlington, VA: Defense Manpower Data Center, 2001), 1-3.
2.
Defense Manpower Data Center, "2006 Survey of Active Duty Spouses: Financial Well-Being and Spouse Employment," note no. 2008-005 (Arlington, VA: Defense Manpower Data Center, 2008), 1.
3.
Pay grade O6 corresponds to the rank of Colonel in the Army, Marine Corps, and Air Force and to the rank of Captain in the Navy.
4.
Defense Manpower Data Center, "2006 Survey of Active Duty Spouses: Administration, Datasets, and Codebook," report no. 2006-034 (Arlington, VA: Defense Manpower Data Center, 2007), 8.
5.
Ibid., 8.
6.
Ibid., 10-11, lists the variables used in defining the reporting domains.
7.
Ibid., 9, lists the variables used in defining the sampling strata.
8.
Jill A. Dever and Robert E. Mason, DMDC Sample Planning Tool (Research Triangle Park, NC: Research Triangle Institute, 2003).
9.
Michael SullivanIII , Statistics: Informed Decisions Using Data, 2nd ed. (Upper Saddle River, NJ: Pearson Prentice Hall, 2007), 482.
10.
William G. Cochran , Sampling Techniques, 3rd ed. (New York: John Wiley, 1977), 89-90.
11.
Ibid., 18-20.
12.
The design effect is the sampling variance for a statistic-as computed under the sample design-divided by the sampling variance that would have been obtained from a simple random sample of the same size. See Robert M. Groves, Floyd J. Fowler, Jr., Mick P. Couper, James M. Lepkowski, Eleanor Singer, and Roger Tourangeau, Survey Methodology (New York: John Wiley, 2004), 334.
13.
Defense Manpower Data Center,2006 Survey of Active Duty Spouses: Administration, Datasets, and Codebook, appendices A and B.
14.
Ibid., appendix C.
15.
Ibid., appendix C.
16.
Ibid., 8.
17.
Robert M. Groves and Emilia Peycheva, "The Impact of Nonresponse Rates on Nonresponse Bias: A Meta-Analysis," Public Opinion Quarterly72 (2008): 167-89.
18.
Maurice C. Bryson, "The Literary Digest Poll: Making of an American Myth," American Statistician30 (1977): 184-85.
19.
Lars R. Bergman and Robert Brage, "Survey Experiences and Later Survey Attitudes, Intentions and Behaviour,"Journal of Official Statistics24 (2008 ): 99-113.
20.
Defense Manpower Data Center, 2006 Survey of Active Duty Spouses: Administration, Datasets, and Codebook, 13.
21.
Groves and Peycheva, "Impact of Nonresponse Rates."
22.
Brenda G. Cox and Steven B. Cohen, Methodological Issues for Health Care Surveys (New York: Marcel Dekker, 1985), 190.
23.
SAS Institute, SAS/STAT® 9.1 User’s Guide: The LOGISTIC Procedure (Book Excerpt) (Cary, NC : SAS Institute, 2008).
24.
The New SAS SURVEYLOGISTIC procedure provides a valid method for performing logistic regression on data from complex survey designs. See Anthony B. An, "Performing Logistic Regression on Survey Data with the New SURVEYLOGISTIC Procedure," in Proceedings of the Twenty-Seventh Annual SAS® Users Group International Conference (Cary, NC : SAS Institute, 2002), paper 258-27.
25.
SAS Institute, SAS/STAT® 9.1 User’s Guide, 2328.
26.
P.N. Peduzzi , J. Concato, E. Kemper, T.R. Holford, and A.R. Feinstein, "A Simulation Study of the Number of Events per Variable in Logistic Regression Analysis ," Journal of Clinical Epidemiology99 (1996 ): 1373-79.
27.
Cox and Cohen, Methodological Issues, 204-5.
28.
Donald B.Rubin, Multiple Imputation for Nonresponse in Surveys ( New York: John Wiley, 1987).
29.
Research Triangle Institute, SUDAAN® User’s Manual Release 8.0 (Research Triangle Park, NC: Research Triangle Institute, 2001).
30.
Kirk M.Wolter, Introduction to Variance Estimation, 2nd ed. (New York: Springer, 2007), 226-34.
31.
International Organization for Standardization, Guide to the Expression of Uncertainty in Measurement Corrected and Reprinted 1995 (Geneva: International Organization for Standardization, 1995), 23.
32.
Willard C. Losinger, "A Look at Raking for Weight Adjustment ," Stats33 ( 2002 ): 8-12.
33.
Among them are Brenda L. Moore, "The Propensity of Junior Enlisted Personnel to Remain in Today’s Army," Armed Forces & Society 28 (2002): 257-78; Kirby L. Bowling, Juanita M. Firestone, and Richard J. Harris, "Analyzing Questions That Cannot Be Asked of Respondents Who Cannot Respond," Armed Forces & Society 31 (2005): 411-37; and Juanita M. Firestone and Richard J. Harris, "Sexual Harassment in the U.S. Military Reserve Component: A Preliminary Analysis," Armed Forces & Society (forthcoming).
