Estimation of single-index models based on boosting techniques

Abstract

In single-index models, the link or response function is not considered as fixed. The data determine the form of the unknown link function. In order to obtain a flexible form of the link function, we specify the link function as an expansion in basis function and propose to estimate parameters as well as the link function by weak learners within a boosting framework. It is shown that the method is a strong competitor to existing methods. The method is investigated in simulation studies and applied to real data.

Keywords

boosting choice of link function p-splines single-index models

Get full access to this article

View all access options for this article.

References

Antoniadis A Gregoire G and McKeague IW 2004 Bayesian estimation in single-index models. Statistica Sinica 14, 1147–64.

Bühlmann P 2006

Boosting for high-dimensional linear models

Annals of Statistics 34, 559–583.

Bühlmann P and Hothorn T 2007) Boosting algorithms: regularization, prediction and model fitting (with discussion). Statistical Science, 22, 477–505.

Bühlmann P and Yu B 2003) Boosting with the L2 loss: regression and classification. Journal of the American Statistical Association, 98, 324–39.

Bühlmann P and Yu B 2006 Sparse boosting. Journal of Machine Learning Research, 7, 1001–24.

Carroll RJ , Fan J , Gijbels I and Wand MP 1997) Generalized partially linear single-index models. Journal of the American Statistical Association, 92, 477–89.

Eilers PHC and Marx BD 1996) Flexible smoothing with B-splines and Penalties. Statistical Science, 11, 89–121.

Friedman JH 2001) Greedy function approximation: a gradient boosting machine. The Annals of Statistics, 29, 1189–232.

Friedman JH and Stützle W 1981) Projection pusuit regression. Journal of the American Statistical Association, 76, 817–23.

10.

Gaiffas S and Lecue G 2007) Optimal rates and adaptations in the single-index model using aggregation. Electronic Journal of Statistics, 1, 538–73.

11.

Hansen M and Yu B 2001) Model selection and minimum description length principle. Journal of the American Statistical Association, 96, 746–74.

12.

Härdle W , Hall P and Ichimura H 1993) Optimal smoothing in single-index models. Annals of statistics, 21, 157–78.

13.

Hastie T and Tibshirani R 1990 Generalized additive models. London: Chapman & Hall.

14.

Hristache M , Juditsky A , Polzehl J and Spokoiny V (2001a) Structure adaptive approach for dimension reduction. Annals of Statistics, 29, 1537–66.

15.

Hristache M , Juditsky A and Spokoiny V (2001b) Direct estimation of the index coefficient in a single-index model. Annals of Statistics, 29, 595–623.

16.

Hurvich CM , Simonoff JS and Tsai C 1998) Smoothing parameter selection in nonparametric regression using an improved Akaike information criterion. Journal of the Royal Statistical Society, Series B, 60, 271–93.

17.

Klein RL and Spady RH 1993) An efficient semiparametric estimator for binary response models. Econometrica, 61, 387–421.

18.

McCullagh P and Nelder JA 1989 Generalized linear models, 2nd edition. New York: Chapman & Hall.

19.

Naik PA and Tsai C-L 2001) Single-index model selection. Biometrika, 88, 821–32.

20.

Penrose KW , Nelson AG and Fisher AG 1985 Generalized body composition prediction equation for men using simple measurement techniques. Medicine and Science in Sports and Exercise, 17, 189.

21.

Polzehl J and Sperlich S 2007 Structural adaptive dimension reduction. WIAS Report 1227, Weierstrass Institute for Applied Analysis and Stochastics, Berlin.

22.

Powell JL , Stock JH and Stoker TM 1989) Semiparametric estimation of index coefficients. Econometrica, 57, 1403–30.

23.

Ruppert D 2002) Selecting the number of knots for penalized splines. Journal of Computational and Graphical Statistics, 11, 735–57.

24.

Schapire RE 1990) The strength of weak learnability. Machine Learning, 5, 197–227.

25.

Stoker TM 1986 Consistent estimation of scaled coefficients. Econometrica 54, 1461–81.

26.

Tutz G and Binder H 2006) Generalized additive modelling with implicit variable selection by likelihood based boosting. Biometrics, 62, 961–71.

27.

Tutz G and Binder H 2007) Boosting ridge regression. Computational Statistics & Data Analysis, 51, 6044–59.

28.

Tutz G and Leitenstorfer F 2007) Generalized smooth monotonic regression in additive modeling. Journal of Computational and Graphical Statistics, 16, 165–88.

29.

Weisberg, S and Welsh AH 1994) Adapting for the missing link. Annals of Statistics, 22, 1674–700.

30.

Xia Y , Tong H , Li WK and Zhu L 2002) An adaptive estimation of dimension reduction. Journal of the Royal Statistical Society, Series B, 64, 363–410.

31.

Yu Y and Ruppert D 2002) Penalized spline estimation for partially linear single-index models. Journal of the American Statistical Association, 97, 1042–54.