Making the Most of Your Research Budget: Efficiency of a Three-Method Measurement Design With Planned Missing Data

Abstract

Planned missing data (PMD) designs are an elegant way to incorporate expensive gold standard methods (e.g., biomarker) and cheaper but systematically biased methods (e.g., questionnaires) in research designs while ensuring high statistical power and low research costs. This article outlines a PMD design with one expensive gold standard and two cheap but biased methods (three-method measurement [3-MM] design). The cost effectiveness of different 3-MM-PMD designs is investigated and compared with the cost effectiveness of corresponding same-price two-method measurement designs using a simulation study. The results underline that PMD designs yield higher statistical power compared with complete data designs in a wide variety of conditions. Adding a second cheap method to the measurement model (i.e., using a 3-MM-PMD design) can increase the statistical power of the research design even further while keeping costs constant, when the additional measure is inexpensive, shares only small amounts of bias variance with the initial cheap measure, and when the gold standard measure is highly expensive compared with the cheap measures. Recommendations as well as a computer program for finding the optimal research design are provided.

Keywords

planned missing data two-method measurement structural equation modeling bias-correction research design

In recent years, many modern measurement methods, such as biomarker analysis, brain imaging, or ambulatory assessment, have been developed for a wide variety of psychological research domains. These methods are usually accepted as highly valid gold standard measures. Unfortunately, these modern methods typically have one significant drawback: They are generally very costly in terms of time, needed expertise, incentivizing participants, price of administration (Graham, 2012), and sometimes invasiveness. Thus, many studies still rely on fairly cheap and easily administered methods (e.g., self-report questionnaires). The problem with these cheap measurement methods, however, is that they generally lack construct validity and lead to biased estimations (Graham, 2012). When researchers are faced with a fixed research budget and the choice between a cheap but biased and an expensive gold standard measure, they traditionally have three options: (a) they assess a few participants using the expensive gold standard only, which can lead to power issues and large standard errors (SEs) due to a small sample size; or (b) they assess many participants using the cheaper method only, which can lead to biased estimates of the studied construct; or (c) they assess participants using the cheap and the expensive method in order to be able to study the relationship between two structurally different methods, which leads to an even greater reduction of sample size.

Two-Method Measurement Design

With the two-method measurement (2-MM) design introduced by Graham, Taylor, Olchowski, and Cumsille (2006) another approach utilizing the concept of planned missing data (PMD) has been proposed. In the 2-MM-PMD design, only a randomly chosen fraction of the participants is assessed using the cheap and the expensive method, whereas the rest of the participants are asked to provide data only on the cheap method. The major advantage of this design is that it can yield more statistical power than only using the expensive measure and to higher construct and criterion-related validity than using the cheap measure alone while having the exact same research costs (Graham et al., 2006). The basic rationale of the 2-MM-PMD design is that the subsample with data on all measures can be used to (a) validly define the studied construct and to (b) control for the systematic bias of the cheap method. Conversely, the participants with data only on the cheap measures help increase the statistical power (through increasing the total sample size) while simultaneously keeping the costs low (Garnier-Villarreal, Rhemtulla, & Little, 2014). Through the random assignment of participants, a predetermined pattern of missing data is created that meets the missing completely at random assumption (Garnier-Villarreal et al., 2014; Graham et al., 2006). This missing completely at random pattern allows unbiased parameter estimation using widely available procedures for handling missing data such as multiple imputation and full information maximum likelihood (Little, Jorgensen, Lang, & Moore, 2014).

Graham et al. (2006) illustrated the 2-MM-PMD model using the measurement of smoking behavior, where smoking behavior is measured using two scales of a self-report questionnaire as the cheap method and two biochemical indicators (carbon monoxide from a breath sample [CO] and cotinine from a saliva sample [Cotinine]) as the expensive gold standard measure. Latent variable structural equation modeling is used to mathematically define this measurement model. The resulting model can be classified as a variant of the correlated-trait-correlated-method-minus-1 [CT-C(M−1)] model introduced by Eid (2000) and Eid, Lischetzke, Nussbeck, and Trierweiler (2003), where the expensive method (i.e., biochemical indicators of smoking) is used as the reference method defining the latent construct (i.e., smoking behavior). To model the systematic bias of the cheap method (i.e., self-report), a latent bias or method factor on which the indicators of the cheap measures load is used (Graham et al., 2006). Such research designs have been widely used without the aspect of planned missingness in multitrait–multimethod research and can be considered a type of bias-correction model (Graham et al., 2006). The bias-correction property, however, only applies to the latent variable measuring the construct and not to the individual scores. Therefore, only group-level relationships can be studied with the 2-MM-PMD design (Rhemtulla & Little, 2012). Even though it is usually used in cross-sectional research, the 2-MM-PMD design can also be adapted for longitudinal research (see Garnier-Villarreal et al., 2014).

By varying the hypothetical prices for the cheap and the expensive measures as well as the underlying population model, Graham et al. (2006) studied the statistical properties of the 2-MM-PMD design in different conditions using a simulation study. To evaluate the 2-MM-PMD design they compared the statistical power of the 2-MM-PMD design with the power of the 2-MM design with complete data (CD), where all participants were assessed using both—the cheap and the expensive—measures. As their outcome measure, they used the SE of the latent regression coefficient between the Smoking factor and an outside criterion (i.e., general health). Thus, they specified a regression, in which the latent variable Smoking was used to predict the latent outcome variable Health, which was measured using four indicators. The full model using the 2-MM-PMD design for the measurement of smoking behavior is depicted in Figure 1. The results of Graham et al. (2006) reveal that in all tested conditions a PMD design could be found that yielded higher power (i.e., smaller SEs for the regression weight) than the corresponding CD design while having the exact same costs. The authors further showed that 2-MM-PMD designs are especially beneficial when the cheap method is highly valid, the cost differential between the expensive and the cheap methods is high and the latent regression coefficient is small (Graham et al., 2006).

Figure 1.

Two-method measurement design for the effect of smoking behavior on general health.

Limitations of the 2-MM Design

The article by Graham et al. (2006) laid important groundwork in investigating the properties of research designs incorporating planned missingness on the measurement methods used. Some issues, however, are not fully addressed in their work. First, Graham et al. (2006) used one expensive and one cheap method in their model. In some research contexts, it might, however, be of interest and beneficial to study the relationships between multiple assessment methods. For example, in a clinical study on autistic spectrum disorder (ASD), one could investigate to what extent the self-reported symptoms differ from symptoms described by parents or teachers (i.e., self-rating and informant rating as two cheap methods) and how these measures relate to an extensive test battery of ASD (for review on ASD assessments, see Falkmer, Anderson, Falkmer, & Horlin, 2013). Since gold standard measures of ASD are very costly, it might be interesting to see whether incorporating more than one cheap method might reduce the number of patients who have to be assessed with the expensive method even more. Hence, a PMD framework with a measurement model incorporating more than two methods is needed in order to be able to cost-efficiently investigate the relationship between the different ASD measures. Second, Graham et al. (2006) only investigated conditions in which the latent correlation of the cheap method (i.e., self-report) and the expensive method (i.e., biochemical markers) was fairly high. In reality, however, it is often not possible to find cheap measures with such high levels of convergent validity with the gold standard. Thus, it is important to investigate, whether PMD designs keep their beneficial properties when cheap methods are used that are not highly valid.

This article addresses these issues by introducing an extension of the 2-MM-PMD design, where a second cheap and systematically biased method is used in the measurement model. Adding another cheap method helps understand the interactions of the studied construct with the methods used as well as the relationships among the methods.

Three-Method Measurement Model

In research domains where an expensive gold standard method as well as at least two fairly valid, but systematically biased cheap methods (e.g., self-report and informant report) are available, it is possible to use a 3-MM-PMD design. The 3-MM-PMD design corresponds to a CT-C(M−1) measurement model for one trait considered (Eid et al., 2003; Eid, Geiser, & Koch, 2016). For each cheap method, at least two indicators, such as items or scales, must be obtained. Analogous to the 2-MM-PMD design, the expensive gold standard method is used as the reference method to define the construct and should thus be a highly valid measure of the construct. For each of the two cheap measures a latent bias factor is specified. These bias factors are by definition uncorrelated with the measured construct and can be interpreted as systematic deviations of the respective cheap method from the expensive method (Eid et al., 2003; Eid et al., 2016; Geiser, Eid, & Nussbeck, 2008). An individual score on the bias factor greater than zero illustrates that the score on the respective cheap method is higher than what would have been expected by the expensive method, negative factor scores indicate the opposite. The two bias factors can be correlated indicating shared systematic bias or method variance of the two cheap methods (Eid et al., 2003; Eid et al., 2016).

There are numerous potential applications of the 3-MM-PMD design in psychological research. Stress, for example, can be measured using self- and informant reports (e.g., parent and child versions of the Responses to Stress Questionnaire; Connor-Smith, Compas, Wadsworth, Thomsen, & Saltzman, 2000), which could be used as cheap but biased methods. As a valid (but expensive) biomarker of long-term stress, cortisol in human hair can be used (e.g., Groeneveld et al., 2013; Russell, Koren, Rieder, & Van Uum, 2012). Another potential application for the 3-MM-PMD design is the measurement of physical activity. In this field, the very expensive Doubly Labelled Water method is the gold standard for measuring energy expenditure in humans (Buchowski, 2014). There are, however, numerous cheaper but flawed self-report questionnaires available to assess physical activity (Sylvia, Bernstein, Hubbard, Keating, & Anderson, 2014). Such self-report measures can easily be adapted to be used as peer report measures (e.g., Chaumeton, Duncan, Duncan, & Strycker, 2011). In addition, 3-MM-PMD designs can be applied in the context of clinical studies. In depression research, many short and inexpensive screening instruments are available. For some of these instruments, corresponding informant versions have been developed (e.g., Geriatric Depression Scale; Brown & Schinka, 2005). A more valid (but also more expensive) structured clinical interview (e.g., SCID-5; First, Williams, Karg, & Spitzer, 2015) could be administered as a gold standard method to a smaller, randomly drawn subsample.

Referring to the example discussed by Graham et al. (2006), the 3-MM-PMD design will in this article be illustrated using the measurement of smoking behavior. In smoking research, several self- and peer report measures as well as highly valid biochemical indicators are available (e.g., Heath et al., 2003; Palmer, Dwyer, & Semmer, 1994; Stacy, Widaman, Hays, & DiMatteo, 1985). Two biochemical indicators (carbon monoxide from a breath sample [CO] and cotinine from a saliva sample [Cotinine]) can be used as a gold standard method to measure the Smoking factor. As cheap methods, two scales of a self-report smoking survey as well as two scales of a peer-report smoking survey can be used. Using the PMD approach, the biochemical measures are only administered to a random subsample of participants, whereas both of the cheap measures (i.e., self- and peer report) are administered to all participants. In the present simulation study, the latent Smoking variable was, analogously to Graham et al. (2006), used in a latent regression analysis to predict the outcome variable Health, which was measured by four manifest variables. The full model using the 3-MM-PMD design for the measurement of smoking behavior is depicted in Figure 2.

Figure 2.

Three-method measurement design for the effect of smoking behavior on general health.

Aim of the Current Study

The aim of the current study is to investigate the statistical properties of 2-MM-PMD and 3-MM-PMD designs in a variety of different conditions and to examine in which conditions it is advisable to integrate a second cheap method in the assessment process.

Method

A simulation study was conducted to assess and compare the cost-effectiveness of the 2-MM-PMD and 3-MM-PMD designs. For comparison reasons, some simulation conditions correspond to those examined by Graham et al. (2006). In addition to Graham et al.’s (2006) study, we also investigated conditions in which the latent correlation of the cheap measure(s) with the expensive measure is only moderately high to examine whether the PMD approach can also be beneficial in cases where no highly valid cheap methods are available.

Population Models

Structurally, the investigated population models correspond to the models depicted in Figures 1 and 2. All variables—manifest and latent—were standardized in all population models. Table 1 depicts the population parameters used in the different conditions of the simulation. Factor loadings and residual variances for all indicators within each measure were assumed to be equal. In all population models the factor loadings λ_h and λ_sb were set to 0.7, which leads to an error variance of the indicators of the expensive method and the outcome measure of σ_εj = 0.51. The reliability of the indicators of the cheap methods was fixed to approximately 0.50 for all simulation cells, which lead to an error variance of approximately σ_εj = 0.5. For the 3-MM designs, three population models, differing in the construct validity of the cheap measures, were investigated: (a) either both cheap measures were valid measures of the construct (Valid) or (b) one cheap measure was valid and the other one rather biased (Mixed) or (c) both cheap measures were rather biased (Biased). The reliable variance of valid cheap measures was operationalized to be composed of 50% construct-relevant variance and 50% systematic bias. This condition resulted in the following factor loadings: λ_sp = 0.5 and λ_bp = 0.5; λ_ss = 0.5 and λ_bs = 0.5. The resulting manifest correlation between an indicator of a biased cheap method and an indicator of the gold standard is 0.35, the corresponding correlation at the latent level is 0.707, which resembles conditions also studied by Graham et al. (2006). The reliable variance of biased cheap methods was set to be composed of 25% construct-relevant variance and 75% systematic bias resulting in the following factor loadings: λ_sp = 0.35 and λ_bp = 0.61; λ_ss = 0.35 and λ_bs = 0.61. The resulting manifest correlation between an indicator of a biased cheap method and an expensive indicator is 0.245, the corresponding correlation at the latent level is 0.495. Thus, the effect of smaller convergent validity between the cheap and the expensive measure can be studied. Accounting for the fact that the two cheap measures can have common or unique systematic bias, two conditions for the latent correlations of the two bias factors (ρ_{Bias-P, Bias-S} = 0.10 and 0.50) were examined. Also, two different values for the latent correlation of the Smoking factor and the Health variable (ρ_{Smoking, Health} = −0.10 and −0.40) were investigated. Since all variables were standardized, this correlation is equal to the regression weight between the Smoking factor and the criterion variable Health. This resulted in 3 (valid, mixed, biased cheap indicators) × 2 (unique, common bias) × 2 (size of regression weight) = 12 population models for the 3-MM designs.

Table 1.

Population Parameters for the Investigated Conditions.

Condition	Factor loadings
	Trait factor			Bias factor			Latent correlations
	Expensive method	Cheap Method 1	Cheap Method 2	Cheap Method 1	Cheap Method 2	Criterion	Bias factors	Trait and criterion
	λ_sb	λ_ss	λ_sp	λ_bs	λ_bp	λ_h	ρ_{Bias-P, Bias-S}	ρ_{Smoking, Health}
3M-Biased-Unique_(.1)	0.7	0.35	0.35	0.61	0.61	0.7	0.1	−0.1
3M-Biased-Common_(.1)	0.7	0.35	0.35	0.61	0.61	0.7	0.5	−0.1
3M-Biased-Unique_(.4)	0.7	0.35	0.35	0.61	0.61	0.7	0.1	−0.4
3M-Biased-Common_(.4)	0.7	0.35	0.35	0.61	0.61	0.7	0.5	−0.4
3M-Mixed-Unique_(.1)	0.7	0.5	0.35	0.5	0.61	0.7	0.1	−0.1
3M-Mixed-Common_(.1)	0.7	0.5	0.35	0.5	0.61	0.7	0.5	−0.1
3M-Mixed-Unique_(.4)	0.7	0.5	0.35	0.5	0.61	0.7	0.1	−0.4
3M-Mixed-Common_(.4)	0.7	0.5	0.35	0.5	0.61	0.7	0.5	−0.4
3M-Valid-Unique_(.1)	0.7	0.5	0.5	0.5	0.5	0.7	0.1	−0.1
3M-Valid-Common _(.1)	0.7	0.5	0.5	0.5	0.5	0.7	0.5	−0.1
3M-Valid-Unique_(.4)	0.7	0.5	0.5	0.5	0.5	0.7	0.1	−0.4
3M-Valid-Common _(.4)	0.7	0.5	0.5	0.5	0.5	0.7	0.5	−0.4
2M-Biased_(.1)	0.7	0.35	—	0.61	—	0.7	—	−0.1
2M-Biased_(.4)	0.7	0.35	—	0.61	—	0.7	—	−0.4
2M-Valid_(.1)	0.7	0.5	—	0.5	—	0.7	—	−0.1
2M-Valid_(.4)	0.7	0.5	—	0.5	—	0.7	—	−0.4

Note. The model names are composed of type of design (three-method measurement design [3M] vs. two-method measurement design [2M]), factor loadings of cheap measures (both cheap measures have high loadings on the Smoking factor [Valid] vs. only one of the cheap measures has high loadings on the Smoking factor [Mixed] vs. both cheap measures have small loadings on the Smoking factor [Biased]), correlation of bias factors (substantial correlation of Bias-P and Bias-S [Common] vs. small correlation of Bias-P and Bias-S [Unique]). The index references the absolute value of the population regression weight. λ_sp, λ_ss, λ_bp, λ_bs, λ_sb, λ_h, ρ_{Smoking, Health}, and ρ _{Bias-P, Bias-S} correspond to the parameters of the models depicted in Figures 1 and 2 for fully standardized manifest and latent variables.

For comparison, four 2-MM models with corresponding factor loadings and effect sizes were defined and analyzed (see Table 1).

Cost-Ratios and Missing Data

The assumed prices for the self-report and the biochemical markers were constructed based on the values used by Graham et al. (2006). The costs for the administration of the self-report was fixed to $7.30. The prices for the biochemical markers were varied, with costs of $16.78 (Condition A), $30 (Condition B), or $73 (Condition C). Thus, three cost-ratios labeled with A, B, C were used to study the 2-MM designs. For the 3-MM designs, the cost of the additional cheap measure, the peer report, was altered between costing just as much as the self-report (Condition 1: $7.30), half as much as the self-report (Condition 2: $3.65), being very cheap (Condition 3: $1.70), or being free (Condition 4: $0). This resulted in 3 (price of biochemical markers) × 4 (price of peer report) = 12 cost-ratios labeled A1 through A4, B1 through B4 and C1 through C4 for the 3-MM designs. In line with Graham et al. (2006), the available research budget was set to $15050. The possible sample size constellations for the different cost-ratios are depicted in Table 2. For cost-ratio “A,” for example, 625 people can be assessed in a complete data (CD) design, where all participants are assessed using the cheap and the expensive method (i.e., 2-MM-CD design) and for cost-ratio “A1,” 479 people can be assessed using the two cheap methods and the expensive method (i.e., 3-MM-CD design). For the planned missing data (PMD) designs, the sample size for the cheap measures was increased by increments of N = 100, the sample size for the expensive biochemical measure was subsequently reduced according to the cost-ratio to keep the research costs constant. For cost-ratio “A,” for example, with each increase of 100 self-reports, 43 fewer biochemical measures can be assessed in the 2-MM-PMD design. For the 3-MM-PMD, each increase of 100 self- and peer reports results in a drop of 87 biochemical measures in cost-ratio “A1.” However, when the peer report gets cheaper, this drop in sample size for the biochemical measures decreases. For cost-ratio “A3,” for example, with each increase of 100 self- and peer reports, only 54 biochemical measures have to be dropped to stay in the research budget.

Table 2.

Configurations of Sample Sizes in the Different Cost-Ratios for a Research Budget of Approximately $15,050.

A1	A2	A3	A4/A	B1	B2	B3	B4/B	C1	C2	C3	C4/C
								171	179	183	187
								200 (166)	200 (176)	200 (182)	200 (186)
				337	368	385		300 (146)	300 (161)	300 (169)	300 (176)
479				400 (307)	400 (356)	400 (382)	403	400 (126)	400 (146)	400 (157)	400 (166)
500 (462)	542	583		500 (258)	500 (319)	500 (352)	500 (380)	500 (106)	500 (131)	500 (145)	500 (156)
600 (375)	600 (505)	600 (575)	625	600 (210)	600 (283)	600 (322)	600 (356)	600 (86)	600 (116)	600 (132)	600 (146)
700 (288)	700 (440)	700 (521)	700 (592)	700 (161)	700 (246)	700 (292)	700 (331)	700 (66)	700 (101)	700 (120)	700 (136)
800 (201)	800 (375)	800 (468)	800 (549)	800 (112)	800 (210)	800 (262)	800 (307)	800 (46)	800 (86)	800 (108)	800 (126)
900 (114)	900 (310)	900 (414)	900 (505)	900 (64)	900 (173)	900 (232)	900 (283)	900 (26)	900 (71)	900 (95)	900 (116)
1,000 (27)	1,000 (244)	1,000 (361)	1,000 (462)	1,000 (15)	1,000 (137)	1,000 (202)	1,000 (258)		1,000 (56)	1,000 (83)	1,000 (106)
	1,100 (179)	1,100 (307)	1,100 (418)		1,100 (100)	1,100 (172)	1,100 (234)		1,100 (41)	1,100 (71)	1,100 (96)
	1,200 (114)	1,200 (253)	1,200 (375)		1,200 (64)	1,200 (142)	1,200 (210)		1,200 (26)	1,200 (58)	1,200 (86)
	1,300 (49)	1,300 (200)	1,300 (331)		1,300 (27)	1,300 (112)	1,300 (185)		1,300 (11)	1,300 (46)	1,300 (76)
		1,400 (146)	1,400 (288)			1,400 (82)	1,400 (161)			1,400 (34)	1,400 (66)
		1,500 (92)	1,500 (244)			1,500 (52)	1,500 (137)			1,500 (21)	1,500 (56)
		1,600 (39)	1,600 (201)			1,600 (22)	1,600 (112)				1,600 (46)
			1,700 (157)				1,700 (88)				1,700 (36)
			1,800 (114)				1,800 (64)				1,800 (26)
			1,900 (70)				1,900 (39)				1,900 (16)
			2,000 (27)				2,000 (15)

Note. Sample sizes for the complete data design are italicized. For planned missing data designs, total sample sizes are presented and corresponding sample sizes for the expensive method are depicted in parenthesis. Cost-ratios A, B, and C correspond to two-method measurement designs. The other cost-ratios correspond to three-method measurement design. The price for the first cheap method (i.e., self-report) is $7.3 in all cost-ratios. The price for the expensive method and for the additional cheap method (i.e., peer report) is as follows (price expensive method, price additional cheap method): A = ($16.78, —), A1 = ($16.78, $7.3), A2 = ($16.78, $3.65), A3 = ($16.78, $1.7), A4 = ($16.78, $0), B = ($30, —), B1 = ($30, $7.3), B2 = ($30, $3.65), B3 = ($30, $1.7), B4 = ($30, $0), C = ($73, —), C1 = ($73, $7.3), C2 = ($73, $3.65), C3 = ($73, $1.7), C4 = ($73, $0).

Through the combination of the 12 population models and the 12 different cost-ratios, 144 simulation cells for the 3-MM designs were defined and analyzed. On top of that, 3 (cost-ratio) × 4 (population models) = 12 2-MM designs were investigated.

Analysis Models

The simulation was carried out in R (version 3.4.1; R Core Team, 2017). For each simulation cell 1,000 data sets with the different sample size constallations depicted in Table 2 were created using the simsem package (version 0.5-13; Pornprasertmanit, Miller, & Schoemann, 2016). Each of these data sets were analyzed in lavaan (version 0.5-23.1097; Rosseel, 2012) using the implemented full information maximum likelihood algorithm. In order to compare the results with those from Graham et al. (2006), the Health variable was not standardized in the analysis model but rather defined by setting the factor loading of the first health item to 1 and freely estimating the variance of the Health variable. The full code for the simulation is provided as supplemental material (all supplemental materials are available in the online version of the article).

Outcomes

Rates of Convergence and Improper Solutions

First, the rates of convergence and proper solutions were examined. Nonconverged replications are those where the lavaan algorithm failed to find model estimates. Improper solutions are replications that resulted in implausible parameter estimates (e.g., negative residual variances). Those cases were removed before the subsequent analyses.

Standard Error Bias

To assess whether the estimated SEs of the regression weight between the Smoking factor and the outcome variable Health are accurate, the standard error bias (seb) was computed using the following formula (adapted from Bandalos & Leite, 2013):

s e b_{i j} = \frac{\sum_{k = 1}^{r} (\frac{S E_{i j k} - S D_{b e t a_{i j}}}{S D_{b e t a_{i j}}})}{r}; - \infty < s e b < \infty,

where $S E_{i j k}$ is the estimated SE of replication k within the population model i and cost-ratio j, $S D_{b e t a_{i j}}$ is the standard deviation of the estimates of the regression coefficient for all proper replications r in population model i and cost-ratio j. Thus, the seb is a standardized index to quantify the averaged accuracy of the estimation of the SEs in all proper replications of a simulation cell. Negative seb values indicate that the computed SEs have on average been underestimated (in relation to the “true” population parameter), positive seb values indicate that the SEs have on average been overestimated. The greater the absolute value of the seb, the greater is the bias in the estimation of the SE. Seb values that are between −0.1 and 0.1 can be considered “acceptable” (Hoogland & Boomsma, 1998).

Power

The SE of the regression coefficient between the Smoking factor and the Health variable is inversely related to the statistical power for testing the regression coefficient. To get an estimate of this SE for each simulation cell, the SEs from all proper replications were averaged within each condition. The averaged SE quantifies the statistical power of the research design in the given condition. In each simulation cell, the sample size constellation returning the smallest averaged SE reveals the research design with the greatest power to test the regression coefficient. To compare the power of the different conditions, we plotted the averaged SEs in relation to the total sample size for all simulation cells. In order to get a numerical indicator about the potential statistical benefit (in terms of power) of the 2-MM-PMD and 3-MM-PMD designs in relation to the corresponding same-cost CD designs, we computed the relative design efficiency parameter RDE_Miss, where

R D E_{M i s s_{i j}} = \frac{\min (S E_{P M D_{i j}})}{S E_{C D_{i j}}}; 0 < R D E_{M i s s_{i j}} < \infty .

For each population model i and cost-ratio j, the proportion of missing data yielding the smallest average SE was determined for the PMD designs. This SE, $\min (S E_{P M D_{i j}})$ , was divided by the average SE of the complementary same-price CD design, $S E_{C D_{i j}}$ . The resulting RDE_Miss represents the relative size of the SE in the best-performing PMD design in relation to the SE of the coresponding CD design. A value of 1 means that the optimal PMD design for a fixed research budget yields the same statistical power as the CD design with identical research costs. RDE_Miss values greater than 1 indicate that the CD design has more statistical power than the optimal PMD design with identical research costs. RDE_Miss values between 0 and 1 illustrate that the optimal PMD design has more statistical power than the complementary same-cost CD design.

To address the question whether a 2-MM or a 3-MM design is superior (in terms of power) in the different conditions, the parameter RDE_3-MM was computed, where

R D E_{3 - M M_{i j}} = \frac{\min (S E_{3 - M M_{i j}})}{\min (S E_{2 - M M_{k l}})}; 0 < R D E_{3 - M M_{i j}} < \infty .

For each 3-MM population model i and cost-ratio j, the averaged SE of the sample size constellation yielding the greatest statistical power, $\min (S E_{3 - M M_{i j}})$ , was divided by the averaged SE of the sample size constellation yielding the greatest power in the corresponding 2-MM design (i.e., same factor loadings and costs for the self-report as well as same regression weight), $\min (S E_{2 - M M_{k l}})$ . For example, for cost-ratio B3 the 3M-Mixed-Common_(.4) model was compared with the 2M-Valid_(.4) model of cost-ratio B. The interpretation of RDE_3-MM values is analogous to the interpretation of RDE_Miss.

Results

Rates of Convergence and Improper Solutions

Convergence rates and the number of improper solutions were highly dependent on the sample size of complete-cases. Among the CD designs, overall 99.9% of the replications converged and 0.1% of replications lead to improper solutions. Of the 2-MM-PMD designs, 99.8% converged and 5.0% lead to improper results. Of the 3-MM-PMD designs, 98.9% converged and 5.5% lead to improper results. The replications of the models with those sample size constellations yielding the greatest power converged in 99.7% of the cases (range: 95.7% to 100%) and yielded proper solutions in 97.7% of the cases (range: 76% to 100%). Detailed convergence rates for all models are presented in the supplemental materials.

Standard Error Bias

For the CD designs, the seb values ranged from −0.084 to 0.021 (M = −0.031, SD = 0.019). For the 2-MM-PMD, designs the seb ranged from −0.480 to 0.04 (M = −0.077, SD = 0.103) and for the 3-MM-PMD designs, from −0.555 to 0.058 (M = −0.066, SD = 0.089). Figure 3 illustrates that the seb drastically decreases when the sample size of complete-cases gets small. Sample sizes of 125 or more for the complete-cases were sufficient to retain acceptable seb values within the described range, except for one case, where the complete-cases sample size was 137 (i.e., 2M-Biased_(.1) in cost-ratio B1). The seb for the conditions with those sample size constellations yielding the greatest power in the different simulation cells ranged from −0.264 to 0.058 (M = −0.040, SD = 0.047). In 13 of those conditions, the seb was smaller than the critical threshold of −0.1. In all of these conditions, the sample size of complete-cases was smaller than 87.

Figure 3.

Seb values for all simulated conditions in relation to the complete-cases sample size.

Design Efficiency of PMD Designs

Figures 4 to 9 present the SEs of the regression coefficient of the Smoking factor predicting the Health factor averaged across all proper replication for the different simulation cells. For those models with high factor loadings of the self-report scales (i.e., λ_ss = 0.5) the averaged SEs decrease at first with an increasing total sample size (or stated differently with a decreasing sample size of complete-cases). After reaching a minimum, the SEs then increase again once the sample size of complete-cases becomes rather small. Conversely, in some of the conditions with low-factor loadings of the self-report scales (i.e., λ_ss = 0.35) the averaged SEs do not decrease initially with increasing total sample sizes. Table 3 illustrates the RDE_Miss values for all conditions. With increasing costs of the biochemical indicators RDE_Miss decreases. Furthermore, RDE_Miss of 3-MM-PMD designs decreases when the cost of the peer report decreases and latent correlation of the two bias factors is small (i.e., unique method variance).

Figure 4.

SEs for the regression weight of the Smoking factor predicting Health for all conditions with a highly valid initial cheap method (i.e., self-report) for cost-ratios A, A1, A2, A3, and A4.

Figure 5.

SEs for the regression weight of the Smoking factor predicting Health for all conditions with an only moderately valid initial cheap method (i.e., self-report) for cost-ratios A, A1, A2, A3, and A4.

Figure 6.

SEs for the regression weight of the Smoking factor predicting Health for all conditions with a highly valid initial cheap method (i.e., self-report) for cost-ratios B, B1, B2, B3, and B4.

Figure 7.

Figure 8.

SEs for the regression weight of the Smoking factor predicting Health for all conditions with a highly valid initial cheap method (i.e., self-report) for cost-ratios C, C1, C2, C3, and C4.

Figure 9.

Table 3.

Relative Design Efficiency of Planned Missing Data Designs Compared With Same-Price Complete Data Designs.

Condition	Cost-ratio
Condition	A/A1	A2	A3	A4	B/B1	B2	B3	B4	C/C1	C2	C3	C4
3M-Biased-Unique_(.1)	1.005	1.004	0.984	0.949	0.976	0.924	0.879	0.831	0.795	0.736	0.691	0.652
3M-Biased-Common_(.1)	1.021	1.010	1.007	0.986	1.005	0.972	0.924	0.879	0.838	0.791	0.727	0.691
3M-Biased-Unique_(.4)	1.002	0.990	0.974	0.955	0.973	0.944	0.911	0.889	0.881	0.822	0.803	0.776
3M-Biased-Common_(.4)	1.002	0.997	0.989	0.974	0.981	0.963	0.938	0.917	0.894	0.859	0.830	0.800
3M-Mixed-Unique_(.1)	0.965	0.902	0.866	0.820	0.863	0.794	0.749	0.703	0.695	0.634	0.582	0.549
3M-Mixed-Common_(.1)	0.988	0.936	0.898	0.852	0.897	0.829	0.783	0.739	0.705	0.658	0.615	0.572
3M-Mixed-Unique_(.4)	0.971	0.937	0.916	0.885	0.910	0.865	0.833	0.805	0.784	0.751	0.717	0.683
3M-Mixed-Common_(.4)	0.985	0.957	0.934	0.906	0.924	0.892	0.859	0.824	0.804	0.762	0.742	0.706
3M-Valid-Unique_(.1)	0.902	0.840	0.801	0.753	0.799	0.730	0.688	0.640	0.606	0.585	0.532	0.499
3M-Valid-Common_(.1)	0.933	0.866	0.827	0.781	0.830	0.763	0.715	0.664	0.657	0.604	0.558	0.522
3M-Valid-Unique_(.4)	0.937	0.902	0.873	0.841	0.857	0.819	0.787	0.750	0.735	0.688	0.661	0.635
3M-Valid-Common_(.4)	0.955	0.920	0.891	0.864	0.884	0.838	0.807	0.778	0.753	0.717	0.680	0.657
2M-Biased_(.1)	1.017	—	—	—	0.980	—	—	—	0.802	—	—	—
2M-Biased_(.4)	0.991	—	—	—	0.955	—	—	—	0.865	—	—	—
2M-Valid_(.1)	0.881	—	—	—	0.758	—	—	—	0.588	—	—	—
2M-Valid_(.4)	0.925	—	—	—	0.843	—	—	—	0.729	—	—	—

Note. The values in the table represent RDE_Miss. The models correspond to the population models described in Table 1 and Figures 1 and 2. The price for the first cheap measure (i.e., self-report) is $7.3 in all cost-ratios. The price for the expensive measure and for the additional cheap measure (i.e., peer report) is as follows (price expensive measure, price additional cheap measure): A = ($16.78, —), A1 = ($16.78, $7.3), A2 = ($16.78, $3.65), A3 = ($16.78, $1.7), A4 = ($16.78, $0), B = ($30, —), B1 = ($30, $7.3), B2 = ($30, $3.65), B3 = ($30, $1.7), B4 = ($30, $0), C = ($73, —), C1 = ($73, $7.3), C2 = ($73, $3.65), C3 = ($73, $1.7), C4 = ($73, $0). All configurations cost approximately $15,050.

Table 4.

Relative Design Efficiency of the Optimal Three-Method Measurement Designs in Comparison With the Corresponding Optimal Same-Cost Two-Method Measurement Designs.

Condition	Cost-ratio
Condition	A1	A2	A3	A4	B1	B2	B3	B4	C1	C2	C3	C4
3M-Biased-Unique_(.1)	1.134	1.066	1.014	0.944	1.082	0.980	0.912	0.843	1.017	0.927	0.862	0.805
3M-Biased-Common_(.1)	1.136	1.072	1.032	0.985	1.113	1.031	0.958	0.898	1.080	0.996	0.902	0.856
3M-Biased-Unique_(.4)	1.136	1.057	1.004	0.949	1.098	1.019	0.966	0.919	1.034	0.957	0.924	0.880
3M-Biased-Common_(.4)	1.140	1.072	1.023	0.973	1.114	1.045	0.997	0.951	1.064	1.005	0.958	0.919
3M-Mixed-Unique_(.1)	1.237	1.092	1.007	0.923	1.225	1.075	0.994	0.916	1.209	1.089	0.987	0.924
3M-Mixed-Common_(.1)	1.277	1.135	1.050	0.965	1.276	1.132	1.045	0.963	1.246	1.141	1.047	0.968
3M-Mixed-Unique_(.4)	1.189	1.078	1.014	0.948	1.161	1.061	0.999	0.939	1.109	1.034	0.987	0.930
3M-Mixed-Common_(.4)	1.209	1.107	1.042	0.976	1.188	1.098	1.032	0.971	1.148	1.065	1.023	0.969
3M-Valid-Unique_(.1)	1.142	1.002	0.922	0.837	1.120	0.975	0.900	0.823	1.106	0.992	0.895	0.827
3M-Valid-Common_(.1)	1.192	1.041	0.960	0.873	1.166	1.026	0.945	0.856	1.144	1.030	0.944	0.869
3M-Valid-Unique_(.4)	1.132	1.024	0.956	0.891	1.085	0.988	0.929	0.870	1.035	0.947	0.895	0.852
3M-Valid-Common_(.4)	1.161	1.052	0.986	0.919	1.125	1.022	0.961	0.905	1.065	0.986	0.934	0.889

Note. The values in the table represent RDE_3-MM. The models correspond to the 3-MM models depicted in Table 1 and Figure 2. These are compared in terms of SE of the regression weight between the Smoking factor and the outcome variable Health with their corresponding 2-MM models. The price for the first cheap measure (i.e., self-report) is $7.3 in all cost-ratios. The price for the expensive measure and for the additional cheap measure (i.e., peer report) is as follows (price expensive measure, price additional cheap measure): A1 = ($16.78, $7.3), A2 = ($16.78, $3.65), A3 = ($16.78, $1.7), A4 = ($16.78, $0), B1 = ($30, $7.3), B2 = ($30, $3.65), B3 = ($30, $1.7), B4 = ($30, $0), C1 = ($73, $7.3), C2 = ($73, $3.65), C3 = ($73, $1.7), C4 = ($73, $0). All configurations cost approximately $15,050.

High Factor Loadings of Self-Report

Figures 4, 6, and 8 illustrate the averaged SEs of the conditions with high factor loadings of the self-report scales on the Smoking factor (i.e., λ_ss = 0.5). RDE_Miss values range from 0.50 to 0.99. 3-MM-PMD models with two valid cheap measures (i.e., λ_{ss =} λ_sp = 0.5) have lower RDE_Miss values than 3-MM-PMD models, in which the validity of the cheap measures is mixed (i.e., λ_ss = 0.5; λ_sp = 0.35). RDE_Miss is smaller for models where the size of the standardized regression effect between the Smoking factor with the Health factor is −.10 rather than −.40.

Low Factor Loadings of Self-Report

Figures 5, 7, and 9 illustrate the averaged SEs of the conditions with low factor loadings of the self-report on the Smoking factor (i.e., λ_ss = 0.35). RDE_Miss values for these conditions range from 0.65 to 1.02. Generally, RDE_Miss values become smaller, when the regression effect between the Smoking factor and the Health factor is −.10 rather than −.40. For cost-ratios A, A1, A2, and A3 (i.e., biochemical indicators are fairly cheap [$16.78]) as well as cost-ratio B1 (i.e., biochemical indicator: $30; peer report and self-report: $7.3) this relationship is reversed.

Design Efficiency of 3-MM Models

Figures 4 to 9 illustrate that the averaged SEs of the 3-MM models with optimal sample size constellations yield larger SEs (i.e., less power) than the corresponding optimal 2-MM designs, when the self-report costs as much as the peer report (i.e., cost-ratios A1, B1, and C1). When the peer report is cheaper (i.e., cost-ratios A3, A4, B2, B3, B4, C2, C3, and C4), the optimal 3-MM designs yield higher power than the corresponding optimal 2-MM designs in some of the population models investigated. Table 4 depicts all RDE_3-MM values.

High Factor Loadings of Self-Report (Figures 4, 6, and 8)

For population models with high factor loadings of the self-report scales on the Smoking factor (i.e., λ_ss = 0.5), RDE_3-MM ranges from 0.82 to 1.28. RDE_3-MM is smaller for models with two valid cheap measures (i.e., λ_ss = λ_sp = 0.5). For models with mixed validity of the cheap measures (i.e., λ_ss = 0.5; λ_sp = 0.35), RDE_3-MM values are only smaller than 1 in cost-ratios A4, B3, B4, C3, and C4. Generally, RDE_3-MM gets smaller when the peer report is cheap and the bias factors are not highly correlated.

Low Factor Loadings of Self-Report (Figures 5, 7, and 9)

For population models with low factor loadings of the self-report scales on the Smoking factor (i.e., λ_ss = 0.35), RDE_3-MM ranges from 0.805 to 1.143. For these conditions, models with smaller population regression weights generally yield smaller RDE_3-MM. In cost-ratio B2, RDE_3-MM is smaller than 1 for the model 3M-Biased-Unique_(.1). In cost-ratios A4, B3, B4, C2, C3, and C4, RDE_3-MM is smaller than 1 in all conditions except for model 3M-Biased-Common_(.4) in cost-ratio C2.

Discussion

A PMD design incorporating two cheap but biased methods and one expensive gold standard method was outlined and illustrated using the example of the hypothetical measurement of smoking behavior. The performance (in terms of power for testing the regression coefficient between the measured construct and an outside criterion) of this 3-MM-PMD design was examined in numerous conditions using a simulation study. Corresponding 2-MM-PMD and CD designs with identical research costs were defined and contrasted with the 3-MM-PMD designs.

Estimation Issues

Convergence rates of the models were generally high, except for some sample size constellations where the number of complete-cases was very small (i.e., N_CC < 125). The seb values indicate that the SEs were generally slightly underestimated in both the 2-MM-PMD and the 3-MM-PMD models. This effect grew when the number of complete-cases got smaller. For those sample size constellations yielding the greatest statistical power for testing the latent regression weight in each condition, the seb fell under the critical cutoff of −0.1 in 13 cases. In all of these instances, the design with the highest power had a rather small sample of complete-cases (i.e., N_CC < 87). This shows that the sample of complete-cases needs to be sufficiently large (e.g., N_CC > 125) to efficiently estimate the 2-MM-PMD and 3-MM-PMD designs. There were, however, no major differences in estimation efficiency between 2-MM-PMD and 3-MM-PMD designs.

Efficiency of PMD Designs

The comparison of the PMD and CD designs in terms of statistical power revealed that PMD designs could increase the cost-effectiveness in almost all simulation conditions compared with the corresponding same-cost CD designs. It is, however, important to differentiate between situations in which the validity of cheap measures is high and when it is rather low.

Cheap Measure(s) Are Highly Valid

In all studied conditions, where at least one of the cheap measures was highly valid (i.e., λ_ss = 0.5), a PMD design could be found that yielded higher statistical power for testing the regression weight than the corresponding same-cost CD design. Higher costs for the expensive gold standard method increased the power benefit of 2-MM-PMD and 3-MM-PMD designs. A small regression weight in the population increased the power benefit even more. Hence, these results replicate the findings by Graham et al. (2006). 3-MM-PMD designs become more cost effective, when the price of the second cheap measure is low, when the second cheap measure is highly correlated with the expensive gold measure and when the two cheap measures share little bias variance. In the most extreme case of the simulation conditions with highly valid cheap measures, the SE of the PMD design with the optimal sample size constellation was roughly half the size of the SE in the corresponding same-price CD design.

Cheap Measures Have Poor Validity

The simulation revealed that using a PMD design in situations in which the cheap measures have poor validity can be statistically harmful in terms of power, when the cost-differential between the expensive measure and the cheap measures is rather small (i.e., cost-ratios A, A1, A2, A3, and B1). However, with an increasing cost differential of the expensive method and the cheap method(s), PMD designs were found that yield smaller SEs (i.e., higher power) compared with the corresponding same-price CD designs. In the most extreme case of the simulation conditions, where the cheap measures had rather poor validity, the SE of the PMD design with a statistically optimal sample size constellation was 65% of the size of the SE in the corresponding same-price CD design. Again, 3-MM-PMD designs yielded more power when the two cheap measures did not share a high amount of bias-related variance and the price of the additional cheap measure was low.

Efficiency of 3-MM Designs

In a second step, we compared the 2-MM and the 3-MM designs among each other in terms of statistical power.

Additional Cheap Method Is Equally Expensive to the Initial Cheap Method

When the additional cheap method in the 3-MM design costs as much as the initial cheap method, the 2-MM design is statistically superior to the 3-MM design in terms of power. With increasing costs of the expensive method, however, this statistical advantage of the 2-MM over the 3-MM design becomes smaller, in some conditions even ignorable. This is especially the case, when the systematic bias variances of cheap methods are unique in the 3-MM design and the additional cheap method is a highly valid measure of the construct.

Additional Cheap Method Is Less Expensive Than the Initial Cheap Method

When an additional cheap measure can be found that has a significantly lower cost of administration than the initial cheap method, 3-MM designs yield higher statistical power than corresponding same-price 2-MM designs, when the two cheap methods are highly valid measures of the construct and only share little amounts of bias variance. This was true for all simulated conditions except for those conditions, in which the administration of the additional cheap method was half as expensive as the administration of the initial cheap method and the price of the expensive measure was fairly low (i.e., cost-ratio A2). Furthermore, 3-MM designs even increased the statistical power relative to the corresponding same-cost 2-MM design when the additional cheap measure was only moderately valid (i.e., λ_sp = 0.35) and the initial cheap measure was highly valid (i.e., λ_ss = 0.5). In our simulation this was, however, only possible, when the two cheap measures did not share a lot of systematic bias variance and the additional cheap measure was very cheap (i.e., 3M-Mixed-Unique_(.1) and 3M-Mixed-Unique_(.4₎ in cost-ratios B3 and C3) or if the additional cheap measure did not add any additional costs to the research design (i.e., all “Mixed” models in cost-ratio A4, B4, and C4). When both cheap methods had poor validity, 3-MM designs were statistically superior to 2-MM designs, when either the additional cheap measure was very cheap or the gold standard method was rather expensive (i.e., cost-ratio A4, B3, B4, C2, C3, and C4).

Recommendations

This article underlines that utilizing a PMD design can help increase the statistical power of a study. For the first time, however, it was also shown that 2-MM-PMD and 3-MM-PMD designs can yield lead less power compared with corresponding CD designs, when the cheap methods used are not highly valid and the cost-differential between the cheap and the expensive methods is small. It is, therefore, important to keep the following recommendations in mind when planning a study with PMD:

2-MM-PMD and 3-MM-PMD designs should only be used when a highly valid gold standard method for measuring the construct of interest is available. Using a systematically biased measure as the reference method would lead to a more difficult interpretation of the construct and the bias factor(s). How the construct and the bias factors can be interpreted is neatly explained by Geiser et al. (2008).

In order for the 2-MM-PMD and 3-MM-PMD models to be identified, at least two observed indicators of the cheap measures (that are systematically biased) and at least one manifest indicator of the expensive measure are needed (Garnier-Villarreal et al., 2014).

The cheap method(s) used should have substantial construct validity (i.e., convergent validity with the gold standard). Highly biased cheap measures can lead to reduced statistical power of the PMD designs compared with same-price CD designs.

High cost differentials between the cheap and the expensive methods as well as a rather small correlation of the construct with the outcome variable increase the statistical benefits of PMD designs.

Sample sizes must be sufficiently large to get stable and efficient model estimates. Our findings underline Rhemtulla and Little’s (2012) suggestion of a minimum of 125 complete-cases.

The present simulation study revealed that incorporating an additional cheap method in a 3-MM-PMD design rather than using a 2-MM-PMD design is recommended, when

the additional cheap measure has high levels of construct validity and can be administered with rather low additional costs.

the gold-standard method is very expensive and the two cheap methods do not share high amounts of method variance.

Finding the Optimal Study Design

The presented findings reveal a complex picture with respect to the properties of 2-MM-PMD and 3-MM-PMD designs and emphasize the importance of the costs and validity of the measures used, the population effect size and the available research budget. Thus, prior to conducting a study, intensive planning is essential to make an informed decision about which research design and sample size constellation to use.

Statistical Considerations

In order to find the research design with the highest power for a given research budget and known assessment methods, it is highly advised to conduct a simulation similar to the one presented in this article. Using shiny (version 1.0.5; Chang, Cheng, Allaire, Xie, & McPherson, 2017), we have developed a program to conduct such calculations for the measurement models used in this article (see Figures 1 and 2). In a first step, the user specifies the model type (2-MM vs. 3-MM), the costs of the methods used and the available research budget using the graphical interface. The program then returns the different sample size constellations. In a second step, the user can run simulations for 2-MM and 3-MM designs to find the research design yielding the greatest power for testing the regression coefficient. For doing so, the population parameters (e.g., factor loadings, latent correlations) need to be specified. When the parameters are chosen reasonably, the program should conduct the simulation in less than 5 minutes. The averaged SEs for the different sample size constellations are graphically illustrated and the design with the best statistical properties in terms of power for testing the regression coefficient is returned. Also, the result tables are available for download in different file types (.Rdata or .csv). The program can be downloaded from https://github.com/MarioLawes/PMD-Simulator or accessed online under https://mariolawes.shinyapps.io/r-shiny-app/. Screenshots of the program are depicted in the supplemental materials.

Practical Considerations

When deciding on which research design to use, the needed sample sizes can be an important factor for the researcher as well. If the expensive method is, for example, rather invasive or has potential negative side effects, PMD designs with a low sample size for the expensive and invasive method (and a high total sample size) can help limit the potentially harmful consequences of a study. However, if recruiting participants is an issue, sample size constellation with rather small total sample sizes (and high amounts of complete-cases) can be beneficial from a practical standpoint. In all cases, at least 125 complete-cases should be available to obtain stable and trustworthy estimates. Thus, depending on the conditions it is important to consider the statistical as well as practical features of the research design.

Conclusions

In this article a range of psychological research domains have been described where highly valid self-report and informant report measures as well as gold standard measures are available. Self- and informant reports are very suitable for 3-MM-PMD designs since they are usually prone to different kinds of response biases and thus share only small amounts of systematic method variance (Paunonen & O’Neill, 2010). Furthermore, the additional assessment of informant reports does not significantly increase recruitment cost for each participant; thus, the administration costs of informant reports is usually pretty low (Vazire, 2006). Also, the costs of modern “objective” gold standard methods are usually very high. For example, from our own research, we know that the analysis of hair cortisol costs around $35 per participant. The administration and scoring of a clinical interview takes at least 2.5 hours and needs to be done by trained interviewers (thus costing at least $50 per participant). The Doubly Labeled Water method even cost around £500 ($680) per participant (Bluck, 2008). Thus, there are many research areas in which the 3-MM-PMD design can be applied effectively.

Limitations and Future Research

Future studies should examine why 3-MM-PMD designs seem to be less cost-effective than 2-MM-PMD design in conditions where the additional cheap measure costs as much as the other cheap measure. This finding could exclusively be due to a reduced sample size for estimating the latent outcome variable (i.e., health) and does not necessarily need to be linked to a less accurate estimation of the construct of interest (i.e., smoking behavior). To investigate this idea, the outcome variable could also be measured using multiple methods (e.g., self- and peer report). Furthermore, to be in line with Graham et al. (2006), the reliabilities of the manifest variables in the presented simulation were assumed to be rather low (Reliability = 0.5). Through systematically varying the reliabilities of the manifest indicators, the effect of the reliability on the cost-effectiveness of the designs could be examined. In addition, the performance of a less restrictive model where the bias factors can correlate with the outcome could be tested. Graham (2012) investigated the performance of such a full bias-correction model for the 2-MM-PMD design and concluded that such a model increases the statistical power in comparison to the same-price CD design only in some (rather extreme) conditions. Similar analyses should be conducted for the 3-MM-PMD design. Also, most of the research on 2-MM-PMD and 3-MM-PMD designs focuses on the SEs of the regression coefficient. Future studies could use different outcomes to get a more complete understanding of the 2-MM-PMD and 3-MM-PMD designs.

In the conducted simulation, the designs have been examined under optimal conditions where the analysis models were equivalent to the population model and no unplanned missing data were present. Thus, it would be worthwhile to study the influence of unplanned missing data in future simulations. Especially when the optimal research design for a fixed budget only has very few complete-cases, the additional unplanned missingness could lead to estimation problems.

Even more important is, however, to examine the performance of the 2-MM-PMD and 3-MM-PMD designs in real-world data sets. To our knowledge, only very few researchers have applied the cross-sectional 2-MM-PMD design to real data (e.g., Zawadzki, Graham, & Gerin, 2012) and only one group of authors have used the 2-MM-PMD design in the data collection process (i.e., Drake et al., 2013). This lack of usage probably resembles the fact that many authors do not know about the benefits of PMD designs or have preconceptions about increased analytical efforts. Also in some research areas, it can be difficult to find an unbiased gold standard, which is essential for the 2-MM-PMD and 3-MM-PMD designs. Still many applications for the 2-MM-PMD design have been proposed (Graham, 2012; Graham et al., 2006; Rhemtulla & Little, 2012). This article additionally presented several hypothetical applications for the 3-MM-PMD design. Additionally, the shiny app accompanying this article provides an easy-to-use tool for planning a study with PMD and may help make 2-MM-PMD and 3-MM-PMD designs more accessible for applied researchers so that they can start to make the most of their research budgets.

Supplemental Material

supplementary_material – Supplemental material for Making the Most of Your Research Budget: Efficiency of a Three-Method Measurement Design With Planned Missing Data

Supplemental material, supplementary_material for Making the Most of Your Research Budget: Efficiency of a Three-Method Measurement Design With Planned Missing Data by Mario Lawes, Martin Schultze and Michael Eid in Assessment

Supplemental Material

Supplementary_Materials – Supplemental material for Making the Most of Your Research Budget: Efficiency of a Three-Method Measurement Design With Planned Missing Data

Supplemental material, Supplementary_Materials for Making the Most of Your Research Budget: Efficiency of a Three-Method Measurement Design With Planned Missing Data by Mario Lawes, Martin Schultze and Michael Eid in Assessment

Footnotes

Acknowledgements

We thank Maximilian Bee and Sören Brähmer for helping to code the shiny app.

Declaration of Conflicting Interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) received no financial support for the research, authorship, and/or publication of this article.

Supplemental Material

The online supplemental material is available at

References

Bandalos

D. L.

Leite

(2013). The use of Monte Carlo studies in structural equation modeling research. In Hancock

G. R.

Mueller

R. O.

(Eds.), Structural equation modeling: A second course (2nd ed., pp. 625-666). Greenwich, CT: Information Age.

Bluck

L. J. C.

(2008). Doubly labelled water for the measurement of total energy expenditure in man: Progress and applications in the last decade. Nutrition Bulletin, 33, 80-90. doi:10.1111/j.1467-3010.2008.00695.x

Brown

L. M.

Schinka

J. A.

(2005). Development and initial validation of a 15-item informant version of the Geriatric Depression Scale. International Journal of Geriatric Psychiatry, 20, 911-918. doi:10.1002/gps.1375

Buchowski

M. S.

(2014). Doubly labeled water is a validated and verified reference standard in nutrition research. Journal of Nutrition, 144, 573-574. doi:10.3945/jn.114.191361

Chang

Cheng

Allaire

Xie

McPherson

(2017). shiny: Web application framework for R. Retrieved from https://cran.r-project.org/package=shiny

Chaumeton

Duncan

S. C.

Duncan

T. E.

Strycker

L. A.

(2011). A measurement model of youth physical activity using pedometer and self, parent, and peer reports. International Journal of Behavioral Medicine, 18, 209-215. doi:10.1007/s12529-010-9118-5

Connor-Smith

J. K.

Compas

B. E.

Wadsworth

M. E.

Thomsen

A. H.

Saltzman

(2000). Responses to stress in adolescence: Measurement of coping and involuntary stress responses. Journal of Consulting and Clinical Psychology, 68, 976-992. doi:10.1037//0022-006X.68.6.976

Drake

K. M.

Longacre

M. R.

Dalton

M. A.

Langeloh

Peterson

K. E.

Titus

L. J.

Beach

M. L.

(2013). Two-method measurement for adolescent obesity epidemiology: Reducing the bias in self-report of height and weight. Journal of Adolescent Health, 53, 322-327. doi:10.1016/j.jadohealth.2013.03.026

Eid

(2000). A multitrait-multimethod model with minimal assumptions. Psychometrika, 65, 241-261. doi:10.1007/BF02294377

10.

Eid

Geiser

Koch

(2016). Measuring method effects: From traditional to design-oriented approaches. Current Directions in Psychological Science, 25, 275-280. doi:10.1177/0963721416649624

11.

Eid

Lischetzke

Nussbeck

F. W.

Trierweiler

L. I.

(2003). Separating trait effects from trait-specific method effects in multitrait-multimethod models: A multiple-indicator CT-C(M-1) model. Psychological Methods, 8, 38-60. doi:10.1037/1082-989X.8.1.38

12.

Falkmer

Anderson

Falkmer

Horlin

(2013). Diagnostic procedures in autism spectrum disorders: A systematic literature review. European Child & Adolescent Psychiatry, 22, 329-340. doi:10.1007/s00787-013-0375-0

13.

First

M. B.

Williams

J. B. W.

Karg

R. S.

Spitzer

R. L.

(2015). Structured clinical interview for DSM-5–Research version (SCID-5-RV). Arlington, VA: American Psychiatric Association.

14.

Garnier-Villarreal

Rhemtulla

Little

T. D.

(2014). Two-method planned missing designs for longitudinal research. International Journal of Behavioral Development, 38, 411-422. doi:10.1177/0165025414542711

15.

Geiser

Eid

Nussbeck

F. W.

(2008). On the meaning of the latent variables in the CT-C(M-1) model: A comment on Maydeu-Olivares and Coffman (2006). Psychological Methods, 13, 49-57. doi:10.1037/1082-989X.13.1.49

16.

Graham

J. W.

(2012). Missing data (1st ed.). New York, NY: New York, NY: Springer. doi:10.1007/978-1-4614-4018-5

17.

Graham

J. W.

Taylor

B. J.

Olchowski

A. E.

Cumsille

P. E.

(2006). Planned missing data designs in psychological research. Psychological Methods, 11, 323-343. doi:10.1037/1082-989X.11.4.323

18.

Groeneveld

M. G.

Vermeer

H. J.

Linting

Noppe

van Rossum

E. F. C.

van IJzendoorn

M. H.

(2013). Children’s hair cortisol as a biomarker of stress at school entry. Stress, 16, 711-715. doi:10.3109/10253890.2013.817553

19.

Heath

A. C.

Knopik

V. S.

Madden

P. A.

Neuman

R. J.

Lynskey

M. J.

Slutske

W. S.

. . . Martin

N. G.

(2003). Accuracy of mothers’ retrospective reports of smoking during pregnancy: Comparison with twin sister informant ratings. Twin Research, 6, 297-301. doi:10.1375/twin.6.4.297

20.

Hoogland

J. J.

Boomsma

(1998). Robustness studies in covariance structure modeling. Sociological Methods & Research, 26, 329-367. doi:10.1177/0049124198026003003

21.

Little

T. D.

Jorgensen

T. D.

Lang

K. M.

Moore

E. W. G.

(2014). On the joys of missing data. Journal of Pediatric Psychology, 39, 151-162. doi:10.1093/jpepsy/jst048

22.

Palmer

R. F.

Dwyer

J. H.

Semmer

(1994). A measurement model of adolescent smoking. Addictive Behaviors, 19, 477-489. doi:10.1016/0306-4603(94)90003-5

23.

Paunonen

S. V.

O’Neill

T. A.

(2010). Self-reports, peer ratings and construct validity. European Journal of Personality, 24, 189-206. doi:10.1002/per.751

24.

Pornprasertmanit

Miller

Schoemann

(2016). simsem: SIMulated Structural Equation Modeling. Retrieved from https://cran.r-project.org/package=simsem

25.

R Core Team. (2017). R: A language and environment for statistical computing. Vienna, Austria: Author.

26.

Rhemtulla

Little

T. D.

(2012). Planned missing data designs for research in cognitive development. Journal of Cognition and Development, 13, 425-438. doi:10.1080/15248372.2012.717340

27.

Rosseel

(2012). lavaan: An R package for structural equation modeling. Journal of Statistical Software, 48(2), 1-36. Retrieved from http://www.jstatsoft.org/v48/i02/

28.

Russell

Koren

Rieder

Van Uum

(2012). Hair cortisol as a biological marker of chronic stress: Current status, future directions and unanswered questions. Psychoneuroendocrinology, 37, 589-601. doi:10.1016/j.psyneuen.2011.09.009

29.

Stacy

A. W.

Widaman

K. F.

Hays

DiMatteo

M. R.

(1985). Validity of self-reports of alcohol and other drug use: A multitrait-multimethod assessment. Journal of Personality and Social Psychology, 49, 219-232. doi:10.1037/0022-3514.49.1.219

30.

Sylvia

L. G.

Bernstein

E. E.

Hubbard

J. L.

Keating

Anderson

E. J.

(2014). Practical guide to measuring physical activity. Journal of the Academy of Nutrition and Dietetics, 114, 199-208. doi:10.1016/j.jand.2013.09.018

31.

Vazire

(2006). Informant reports: A cheap, fast, and easy method for personality assessment. Journal of Research in Personality, 40, 472-481. doi:10.1016/j.jrp.2005.03.003

32.

Zawadzki

M. J.

Graham

J. W.

Gerin

(2012). Increasing the validity and efficiency of blood pressure estimates using ambulatory and clinic measurements and modern missing data methods. American Journal of Hypertension, 25, 764-769. doi:10.1038/ajh.2012.40

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

0.42 MB

0.77 MB