Monday, January 10, 2011

Least Angle Regression and LASSO

Least Angle Regression (LAR)

Similar to ridge regression, LAR is a shrinkage estimator. It is similar to forward selection, but only enters 'as much' of the β estimate as necessary. The β estimate is increased with each iteration of the algorithm, approaching the least squares estimate of β.

1) Start with predictors near 0, let r = yi yest
2) Find  Xi most correlated with r.
3) Move βest -->  βls   until  corr (Xi, r) = corr(Xj, r) for some Xj
4) Continue until all X's have been entered

If  μk = Xβ, then LAR finds μk that makes the smallest angle between each of the predictors and r.


LAR, as a shrinkage estimator, minimizes a penalized residual sum of squares:

min{e'e + λβ) 

where λ is a penalty or shrinkage factor.

Least Absolute Shrinkage and Selection Operator (LASSO)

In essence, with LASSO we have a LAR estimator that is penalized such that |β| < c where c is some constant. 

If we minimize the following penalized residual sum of squares:

min{e'e + λ|β|)

then we obtain the LASSO estimator.

See also Comparing OLS, Ridge Regression, LAR and LASSO. 

References:

The Elements of 
Statistical Learning:
Data Mining, Inference, and Prediction.
Second Edition (link)


No comments:

Post a Comment