## Monday, January 10, 2011

### Least Angle Regression and LASSO

Least Angle Regression (LAR)

Similar to ridge regression, LAR is a shrinkage estimator. It is similar to forward selection, but only enters 'as much' of the β estimate as necessary. The β estimate is increased with each iteration of the algorithm, approaching the least squares estimate of β.

2) Find  Xi most correlated with r.
3) Move βest -->  βls   until  corr (Xi, r) = corr(Xj, r) for some Xj
4) Continue until all X's have been entered

If  μk = Xβ, then LAR finds μk that makes the smallest angle between each of the predictors and r.

LAR, as a shrinkage estimator, minimizes a penalized residual sum of squares:

min{e'e + λβ)

where λ is a penalty or shrinkage factor.

Least Absolute Shrinkage and Selection Operator (LASSO)

In essence, with LASSO we have a LAR estimator that is penalized such that |β| < c where c is some constant.

If we minimize the following penalized residual sum of squares:

min{e'e + λ|β|)

then we obtain the LASSO estimator.