Robustness Regions for Dichotomous Decisions

This expository paper intends to give educational researchers suitable access to a method for investigating the stability of their dichotomous decisions under changes of assumptions. Inferences and decisions based on statistical models typically involve model assumptions, such as normality and independence of observations. In statistical decision theory, one also has to specify loss functions, and in a Bayesian analysis a prior distribution has to be specified. In the case of dichotomous decisions (passing or failing a student, choosing between two teaching methods, rejecting or retaining a hypothesis), the total set of all such assumptions/specifications for which the decision would have been the same is the robustness region (section 2). Inspection of this (data-dependent) region is a form of sensitivity analysis which may lead to improved decision making. Section 1 discusses earlier forms of sensitivity or robustness analysis, both data-dependent and a priori. Examples of robustness regions deal with mastery decisions (sections 3 and 4), evaluation of a teaching experiment (section 5), and aptitude treatment interaction (section 6).