Solutions for Determining the Significance Region Using the Johnson-Neyman Type Procedure in Generalized Linear (Mixed) Models

Abstract

Researchers often compare the relationship between an outcome and covariate for two or more groups by evaluating whether the fitted regression curves differ significantly. When they do, researchers need to determine the “significance region,” or the values of the covariate where the curves significantly differ. In analysis of covariance (ANCOVA), the Johnson-Neyman procedure can be used to determine the significance region; for the hierarchical linear model (HLM), the Miyazaki and Maier (M-M) procedure has been suggested. However, neither procedure can assume nonnormally distributed data. Furthermore, the M-M procedure produces biased (downward) results because it uses the Wald test, does not control the inflated Type I error rate due to multiple testing, and requires implementing multiple software packages to determine the significance region. In this article, we address these limitations by proposing solutions for determining the significance region suitable for generalized linear (mixed) model (GLM or GLMM). These proposed solutions incorporate test statistics that resolve the biased results, control the Type I error rate using Scheffé’s method, and uses a single statistical software package to determine the significance region.

Keywords

generalized estimating equations hierarchical linear models longitudinal analysis multiple testing simultaneous testing

Get full access to this article

View all access options for this article.

References

Coleman

Hoffer

Kilgore

(1982). High school achievement: Public, Catholic, and other schools compared. New York, NY: Basic Books.

Dempster

A. P.

Rubin

D. B.

Tsutakawa

R. K.

(1981). Estimation in covariance components model. Journal of the American Statistical Association, 76, 341–353.

Hunka

(1995). Identifying regions of significance in ANCOVA problems having nonhomogeneous regressions. British Journal of Mathematical and Statistical Psychology, 48, 161–188.

Hunka

Leighton

(1997). Defining Johnson-Neyman regions of significance in the three-covariate ANCOVA using Mathematica. Journal of Education and Behavioral Statistics, 22, 361–387.

Johnson

P. O.

Fay

L. C.

(1950). The Johnson-Neyman technique, its theory and application. Psychometrika, 15, 349–367.

Johnson

P. O.

Neyman

(1936). Tests of certain linear hypothesis and their application to some educational problems. Statistical Research Memoirs, 1, 57–93.

Kenward

M. G.

Roger

J. H.

(1997). Small sample inference for fixed effects from restricted maximum likelihood. Biometrics, 53, 993–997.

Littell

R. C.

Milliken

G. A.

Stroup

W. W.

Wolfinger

R. D.

Schabenberger

(2006). In SAS for mixed models second edition. Cary, NC: SAS Institute, p. 756.

McCullagh

Nelder

J. A.

(1989). Generalized linear models (2nd ed.). London: Chapman and Hall.

10.

McCulloch

C. E.

Searle

S. R.

(2001). Generalized, linear and mixed models New York, NY: John Wiley.

11.

Miyazaki

Maier

K. S.

(2005). Johnson-Neyman type technique in hierarchical linear models. Journal of Education and Behavioral Statistics, 30, 233–259.

12.

Molenberghs

Verbeke

(2004). Meaningful statistical model formulations for repeated measures. Statistica Sinica, 14, 989–1020.

13.

Molenberghs

Verbeke

(2005). Models for discrete longitudinal data. New York: Springer, p. 41.

14.

Potthoff

R. F.

(1964). On the Johnson-Neyman technique and some extensions thereof. Psychometrika, 29, 241–256.

15.

Raudenbush

S. W.

Bryk

A. S.

Cheong

Y. F.

Congdon

(2000). HLM5—Hierarchical linear and nonlinear modeling Chicago, IL: Scientific Software International.

16.

Raudenbush

S. W.

Bryk

A. S.

(2002). Hierarchical linear models: Applications and data analysis methods (2nd ed.). Newbury Park, CA: Sage.

17.

Schaalje

B. G.

McBride

J. B.

Fellingham

G. W.

(2001). Approximation to distributions of test statistics in complex mixed linear models using SAS Proc MIXED. Proceedings of the Twenty-Sixth Annual SAS Users Group International Conference, SAS Institute.

18.

Scheffé

(1959). The analysis of variance New York, NY: Wiley.

19.

Snedecor

W. G.

Cochran

W. G.

(1989). Statistical methods (8th ed.). Ames: Iowa State University Press.

20.

Steffens

F. E.

(1968). On comparing two simple linear regression lines. South African Statistical Journal, 2, 33–53.

21.

Steffens

F. E.

(1971). On confidence sets for the ratio of two normal means. South African Statistical Journal, 5, 105–113.

22.

Zerbe

G. O.

(1978). On Fieller’s theorem and the general linear model. The American Statistician, 32, 103–105.

23.

Zerbe

G. O.

Archer

P. G.

Banchero

Lechner

A. J.

(1982). On comparing regression lines with unequal slopes. American Journal of Physiology: Regulatory, Integrative and Comparative Physiology, 242, 178–180.