Least Angle Regression and LASSO

Least Angle Regression (LAR)

Similar to ridge regression, LAR is a shrinkage estimator. It is similar to forward selection, but only enters 'as much' of the β estimate as necessary. The β estimate is increased with each iteration of the algorithm, approaching the least squares estimate of β.

1) Start with predictors near 0, let r = y_i-  y^est
2) Find  Xⁱ most correlated with r.
3) Move β_est -->  β_ls   until  corr (Xⁱ, r) = corr(X^j, r) for some X^j
4) Continue until all X's have been entered

If  μ_k = Xβ, then LAR finds μ_k that makes the smallest angle between each of the predictors and r.

LAR, as a shrinkage estimator, minimizes a penalized residual sum of squares:

min{e'e + λβ) 

where λ is a penalty or shrinkage factor.

Least Absolute Shrinkage and Selection Operator (LASSO)

In essence, with LASSO we have a LAR estimator that is penalized such that |β| < c where c is some constant. 

If we minimize the following penalized residual sum of squares:

min{e'e + λ|β|)

then we obtain the LASSO estimator.

See also Comparing OLS, Ridge Regression, LAR and LASSO. 

References:

The Elements of 
Statistical Learning:
Data Mining, Inference, and Prediction.
Second Edition (link)