Intuition for Random Effects

Previously I wrote a post based on course notes from J.Blumenstock that attempted to provide some intuition for how fixed effects estimators can account for unobserved heterogeneity (individual specific effects).

Recently someone asked if I could provide a similarly motivating and intuitive example regarding random effects. Although I was not able to come up with a new example, I can definitely discuss random effects in the same context of the previous example. But first a little (less intuitive) background.

Background

To recap, the purpose of both fixed and random effects estimators is to model treatment effects in the face of unobserved individual specific effects.

y_it =b x_it + α_i + u_it  (1) 

In the model above this is represented by α_{i .} In terms of estimation, the difference between fixed and random effects depends on how we choose to model this term. In the context of fixed effects it can be captured through a dummy variable estimation (this creates different intercepts or shifts capturing specific effects) or by transforming the data, subtracting group (fixed effects) means from individual observations within each group.  In random effects models, individual specific effects are captured by a composite error term (α_i + u_it) which assumes that individual intercepts are drawn from a random distribution of possible intercepts. The random component of the error term α_i captures the individual specific effects in a different way from fixed effects models. 

As noted in another post, Fixed, Mixed, and Random Effects, the random effects model is estimated using Generalized Least Squares (GLS) :

β_GLS = (X’Ω^-1X)^-1(X’Ω^-1Y) where Ω = I ⊗ Σ    (2) 

Where Σ is the variance α_i+ u_it . If  Σ is unknown, it is estimated, producing a feasible generalized least squares estimate β_FGLS

Intuition for Random Effects

In my post Intuition for Fixed Effects I noted: 

"Essentially using a dummy variable in a regression for each city (or group, or type to generalize beyond this example) holds constant or 'fixes' the effects across cities that we can't directly measure or observe. Controlling for these differences removes the 'cross-sectional' variation related to unobserved heterogeneity (like tastes, preferences, other unobserved individual specific effects). The remaining variation, or 'within' variation can then be used to 'identify' the causal relationships we are interested in."

Lets look at the toy data I used in that example. 

The crude ellipses in the plots above (motivated by the example given in Kennedy, 2008) indicate the data for each city and the the 'within' variation exploited by fixed effects models (that allowed us to correctly identify the correct price/quantity relationships expected in the previous post). The differences between the ellipses represents 'between variation.' As Kennedy discusses, random effects models differ from fixed effects models in that they are able to exploit both 'within' and 'between' variation, producing an estimate that is a weighted average of both kinds of variation (via Σ in equation 2 above). OLS, on the other hand exploits both kinds of variation as an unweighted average.

More Details 

As Kennedy discusses, both FE and RE can be viewed as running OLS on different transformations of the data.

For fixed effects: "this transformation consists of subtracting from each observation the average of the values within its ellipse"

For random effects: "the EGLS (or FGLS above) calculation is done by finding a transformation of the data that creates a spherical variance-covariance matrix and then performing OLS on the transformed data."

As Kennedy notes, the increased information used by RE makes them more efficient estimators, but correlation between 'x' and the error term creates bias. i.e. RE assumes that α_i is uncorrelated with (orthogonal to) regressors. Angrist and Pischke (2009) discuss (footnote, p. 223) that they prefer FE because the gains in efficiency are likely to be modest while the finite sample properties of RE may be worse. As noted on p.243 an important assumption for identification in FE is that the most important sources of variation are time invariant (because information from time varying regressors gets differenced out). Angrist and Pischke also have a nice discussion on page 244-245 discussing the choice between FE and lagged dependent variable models.

References:

A Guide to Econometrics. Peter Kennedy. 6th Edition. 2008
Mostly Harmless Econometrics. Angrist and Pischke. 2009

See also: ‘Metrics Monday: Fixed Effects, Random Effects, and (Lack of) External Validity (Marc Bellemare.

Marc notes: 

"Nowadays, in the wake of the Credibility Revolution, what we teach students is: “You should use RE when your variable of interest is orthogonal to the error term; if there is any doubt and you think your variable of interest is not orthogonal to the error term, use FE.” And since the variable can be argued to be orthogonal pretty much only in cases where it is randomly assigned in the context of an experiment, experimental work is pretty much the only time the RE estimator should be used."

Intuition for Fixed Effects

I've written about fixed effects before in the context of mixed models. But how are FE useful in the context of causal inference? What can we learn from a panel data using FE that we can't get from a standard regression with cross sectional data?  Let's view this through a sort of parable, based largely on a very good set of notes produced by J. Blumenstock, used in a management statistics course (link).

Suppose we have a restaurant chain and have gathered some cross sectional data on the pricing and consumption of large pizzas for some portion of the day for some period 1 across three cities, as pictured below:

Now, if we are trying to infer the relationship between price and quantity demanded using this data, we notice something odd. The theoretically implied negative relationship does not exist. In fact, if we plot the points, this seems more in line with a supply curve rather than a demand curve:

What's going on that could explain this? One explanation could be specific individual differences across cities related to taste and quality. Perhaps in Chicago, customer's tastes and preferences are for much more expensive and higher quality pizza, and they really like pizza a lot. They may be willing to pay more for more pizzas aligned with their specific tastes and preferences. Perhaps this is also true for San Francisco, but to a lesser extent, and in Atlanta maybe not so much.

What we have is unobserved heterogeneity related to these specific individual effects. How can we account for this? Suppose we instead collected the same data for two periods, essentially creating a panel of data for pizza consumption:

Now, if we look 'within' each city, the data reveals the theoretically implied relationship between price and demand. Take San Francisco for example:

This is essentially what fixed effects estimators using panel data can do. They allow us to exploit the 'within' variation to 'identify' causal relationships. Essentially using a dummy variable in a regression for each city (or group, or type to generalize beyond this example) holds constant or 'fixes' the effects across cities that we can't directly measure or observe. Controlling for these differences removes the 'cross-sectional' variation related to unobserved heterogeneity (like tastes, preferences, other unobserved individual specific effects). The remaining variation, or 'within' variation can then be used to 'identify' the causal relationships we are interested in.

See also: Difference-in-Difference models. These are a special case of fixed effects also used in causal inference.

Reference:
Fixed Effects Models(Very Important Stuff)
www.jblumenstock.com/courses/econ174/FEModels.pdf

Topics Related to Linear Models

While attending a session at this year’s SAS Global Forum, I was reminded of several topics that I have not covered in previous posts, but should have.

Fixed and Random Effects Models and Panel Data (see also Mixed, Fixed and Random Effects)

Notes: Panel data, or repeated measures data, is characterized by multiple observations on individuals over time.  A consequence of this is that measurements on the same individual will likely be more correlated than measurements for (or between) different individuals.  Measurements taken closer together over time will also likely be more correlated than measurements taken at greater intervals. As a result, assumptions from OLS regression regarding independence and homogenous variance will likely be violated.  As noted during the session I attended, in SAS, PROC MIXED appropriately handles within subject and time dependent correlations and the covariance structure associated with repeated measures.

Let’s structure the model as follows: Yit = Xit + Ait +Uit

Where Ait is a unobserved individual effect. There are two assumptions that we can work under when estimating this model,

1)Random Effects (RE): Assumes that Ait is independent of X . X may be considered a  predetermined, or other fixed effect. Ait is a random effect.

Estimation: Feasible Generalized Least Squares-  B_fgls = (X’W^-1X)^-1X’W^-1y

i.e V(e) != σ²  I  which would be the case under OLS

2)Fixed Effects (FE): Assumes Ait is not independent of X.

Estimation: differencing or subtracting the respective means from X and Y  and running OLS on the adjusted data produces the estimate of B.

You could also subtract the lagged version of each variable X and Y from itself respectively to move the time-correlated component, or you could remove Ait through dummy variable regression.

Fixed and Random Effects in Analysis of Variance (in general)

In my previous posts related to AOV and Mixed Models,  I did not discuss these in the context of  AOV.

Typically fixed effects are repeatable factors that are set by the experimenter,  often the ‘treatments.’  Random effects are effects that are selected randomly from a population, often ‘blocks.’

Type I and III Sums of Squares

Type I Sums of Squares: often referred to as sequential sums of squares, this is the SS for each effect adjusted for all effects that appear earlier in the model.

Example:

                                   

Y1 = X1B1                       

Type I SS = SSR(Y1) = SS(x1)

Y2 = X1B1 + X2B2

Type I SS = SSR(Y2) – SSR(Y1) = SS(X2)

Type III sums of squares are adjusted for every X in the full model.

Example:

Y1 = B1X1 + B2X2

Y2 = B1X1

Type III SS = SSR(Y1) – SSR(Y2) = SS(X1)

Y3 = B2X2

Type III SS = SSR(Y1) – SSR(Y3) = SS(X2)

Means vs. LS Means

Means = overall mean for a treatment or factor level

LS Means = within group means adjusted for other effects in the model

Type I SS are used to test differences between means while type III SS are used to test differences between LS Means. 

Mixed, Fixed, and Random Effects Models

The General Mixed Model:

Y  = Xβ + Z α+ e            (1)

β =  fixed effects co-efficient vector

X = fixed effects model matrix

α = random effects co-efficient matrix

Z =  random effects model matrix

The fixed part of the model is specified by Xβ and the random part by Zα+ e.

 The ‘random effects’ matrix (α) represents random effects that vary across individuals vs. the ‘fixed effects’ matrix (β) that represents effects that are the same across all individuals.

Panel Data: Cross sectional time series data, in most cases looking at hundreds or thousands of individuals (units) observed at several points across time, i.e. multiple observations per unit across time.

Heterogeneity Effect: Whether or not effects, or responses of individuals are the same across time, or if there are group differences.  If effects are not the same, and they are not accounted for, estimation errors result. Fixed and random effects models attempt to capture the heterogeneity effect. 

Given

y_it =b x_it + α_i + u_it                  (4)

 α_i is an unobserved individual effect

Fixed Effects Model (FE): α_i is correlated with x. In this model individual effects, or differences across individuals can be captured by shifts in the regression equation, or dummy variables.

As such we can estimate the fixed effects model as a Least Squares Dummy Variable model (LSDV):

y= Xb + dα + e            (5)

where d is a vector of dummy variables for each individual or unit effect.

Random Effects Model (RE): α_i is uncorrelated with x. Individual effects are randomly distributed across units.

y_it =b x_it + α_i+ u_it            (6)

Note that (4) and (6) are identical, the only difference is the assumptions about the individual effect α_i and how the models are estimated.

Note also that (6) differs from (1) in that we don’t have the matrix Z. The Random Effects model is in fact a special case of the general mixed model with a random intercept (α_i).

The random effects model is estimated using Generalized Least Squares (GLS) :

β_GLS = (X’Ω^-1X)^-1(X’Ω^-1Y) where Ω = I ⊗ Σ    (7)

Where Σ is the variance α_i+ u_it . If  Σ is unknown, it is estimated, producing a feasible generalized least squares estimate β_FGLS

Whether a FE or RE model should be used can be determined based on the Housman test (see Greene, for more details)

References:

Greene, Econometric Analysis, 5^th Edition