Sample size and design of experiment issues in testing offers

This article addresses test and experimental design issues in the context of testing offers. Using a simple example, we discuss sample size issues first. In this context, we introduce the problem of multiple comparisons and discuss the Bonferroni, Tukey, and Bechhofer approaches. This leads us to conclude that a “magical” statistically significant sample size does not exist. Rather, the appropriate sample size depends on what questions we want answered as a result of the test. Further, we show the need to identify explicitly all such questions prior to running the test, and that “data snooping” after test results are in may increase the risk of incorrect conclusions or uncertain results. The latter part of the article discusses modeling the offers’ response rates using the logit and probit regression models. We first illustrate this using a full factorial design, and when no interaction effects are found, present a solution based on a one-half fractional factorial design. Though none of this material is totally new to academic researchers, the ideas presented here are not well known in the direct marketing community. Thus, the objective is educational: to build a bridge between the academic and the practitioner communities.