Mediators and Moderators

Moderators: With moderation, a third variable impacts or interacts with the relationship between two other variables. We would say the relationship between two variables is ‘moderated.’ This can be thought of as an interaction in a standard regression:

Y = b0 + b1*X1 + b2*X2 + b3*X1*X2

b3 =moderating effect i.e. the relationship between Y and X1 changes with levels of X2
b1 = impact of X1 on Y when X2 = 0.

So in the context of the relationship between Y and X1, X2 is a moderator

Mediators: With mediation, a third variable invervenes in the relationship between two other variables. For example, in the diagram below, suppose we are interested in the relationship between x and y. This relationship may be ‘mediated’ by a third variable m.

Consider a model where Y = grade in course (our outcome of interest), k = IQ, m =  study skills. We might hypothesize that study skills ‘mediate’ the effect of IQ on course grade. A perfectly brilliant person might do OK on an exam through educated guesses, but we all might know of cases where brilliant students have done quite poor due to lax study skills. So while there may be a direct effect of IQ on grades, IQ -> grades or x -> y there is an indirect effect as well, IQ->Study Skills -> grade or x -> m -> y.

This implies that mediation can take a number of forms and can be formally tested. In the case of full mediation, the relationship between x and y becomes insignificant after a mediator ‘m’ is included in the model, or our estimate of c (modeling the path or direct effect between x and y) isn’t significantly different from 0. Partial mediation would occur if the relationship between x and y or  c is reduced (but remains significant) after m is entered into the model).  In this case we could say that x has both direct effects on y (through the path c) as well as an indirect effect (through the mediator m or paths a and b).

These relationships can be formally tested as laid out in Hair et al:

1) Test for significant correlations between x,y or estimate c; x,m or estimate a; m,y or estimate b
2) If c is significant after m is included, and the magnitude of c does not change then m is not a mediator.
3) If the magnitude of c is reduced after including m, and c remains significant, then m is a moderator. This is a case of partial mediation.
4) If including m in the model reduces the magnitude of c such that it is not significantly different from 0, then m is a mediator and this is considered a case of full mediation.

Reference: Multivariate Data Analysis. 6th Edition. Harris, Black, Babin, Anderson and Tatham. Pearson-Prentice Hall. 2006.

Identification and Common Trend Assumptions in Difference-in-Differences for Linear vs GLM Models

In a previous post I discussed the conclusion from Lechner’s paper 'The Estimation of Causal Effects by Difference-in-Difference Methods', that difference-in-difference models in a non-linear or GLM context failed to meet the common trend assumptions, and therefore failed to identify treatment effects from a selection on unobservables context.

In that post I noted that Lechner points out (quite rigorously in the context of the potential outcomes framework):

"We start with a “natural” nonlinear model with a linear index structure which is transformed by a link function, G(·), to yield the conditional expectation of the potential outcome.....The common trend assumption relies on differencing out specific terms of the unobservable potential outcome, which does not happen in this nonlinear specification... Whereas the linear specification requires the group specific differences to be time constant, the nonlinear specification requires them to be absent. Of course, this property of this nonlinear specification removes the attractive feature that DiD allows for some selection on unobservable group and individual specific differences. Thus, we conclude that estimating a DiD model with the standard specification of a nonlinear model would usually lead to an inconsistent estimator if the standard common trend assumption is upheld. In other words, if the standard DiD assumptions hold, this nonlinear model does not exploit them (it will usually violate them). Therefore, estimation based on this model does not identify the causal effect”

I wanted to review at a high level exactly how he gets to this result. But I wanted to simplify this as much as possible and start with some basic concepts. Starting with a basic regression model, the population conditional expectation function, or conditional mean of Y given X can be written as:

Regression and Expected Value Notation:

E[Y|X] = β₀ + β₁ X (1)

and we estimate this with the regression on observed data:

y = b₀ + b₁X + e (2)

Where b₁ is our estimate of the population parameter of interest β₁.

If E[b₁] = β₁ then we say our estimator is unbiased.

Potential Outcomes Notation:

When it comes to experimental designs, we are interested in knowing counterfactuals, that is what value of an outcome would a treatment or program participant have in absence of treatment (the baseline potential outcome) vs. if they participated or were treated? If we specify these 'potential outcomes' as follows:

Y⁰= baseline potential outcome
Y¹= potential treatment outcome

We can characterize the treatment effect as:

E[Y⁰-Y¹] or the difference in potential treated vs baseline outcomes. This is referred to as the average treatment effect or ATE. Sometimes we are interested in, or some models estimate, the average treatment effect on the treated or ATT :E[Y⁰-Y¹ | d = 1]  

where d is an indicator for treatment (d = 1) vs control or untreated (d =0).

Difference-in-Difference Analysis:

Difference-in-difference (DD) estimators assume that in absence of treatment the difference between control (B) and treatment (A) groups would be constant or ‘fixed’ over time. Treatment effects in DD estimators are derived by subtracting differences between pre and post values within treatment and control groups, and then taking a difference in differences between treatment and control groups. The unobservable effects that are constant or fixed over time 'difference out' allowing us to identify treatment effects controlling for these unobservable characteristics with out explicitly measuring them. This characterizes what is referred to as a 'selection on unobservables' framework.

  This can also be estimated using linear regression with an interaction term:

y = b₀ + b₁ d + b₂ t + b₃ d*t+ e (3)

where d indicates treatment (d=1 vs d = 0) and the estimated coefficient (b₃ ) on the time by treatment interaction term gives us our estimate of treatment effects. 

Lechner and Potential Outcomes Framework:

In an attempt to present the issues with GLM DD models depicted in Lechner (2010) using the simplest notation possible (abusing notation slightly and perhaps at a cost of precision), we can depict the framework for difference-in-difference analysis using expectations:

DID = [E(Y₁|D=1)-E( Y₀|D=1)] -[E(Y₁|D=0)-E(Y₀|D=0)] (4)

DID = [pre/post differences for treatment group] – [pre/post differences for control group]

where Y represents the observed outcome values sub-scripted by pre (0) and post periods(1)

We can represent potential outcomes in the regression framework as follows:

E(Y_t¹|D) = α + tδ¹ + dγ “potential outcome if treated” (5)

E(Y_t⁰|D) = α + tδ⁰ + dγ “potential baseline outcome” (6)

ATET: E(Y_t¹- Y_t⁰|D= 1) = θ₁ = δ    (7)

“difference-in-difference of potential outcomes across time if treated”

We can estimate δ with a regression on observed data of the form:

y = b₀ + b₁ d + b₂ t + b₃ d*t+ e (3')

where b₃ is our estimator for δ.

Common Trend Assumption:

Difference-in-difference (DD) estimators assume that in absence of treatment the difference between control (B) and treatment (A) groups would be constant or ‘fixed’ over time. This can be represented geometrically in a linear modeling context by 'parallel trends' in outcome levels between treatment and control groups in absence of a treatment:

As depicted above, BB represents the trend in outcome Y for a control group. AA represents the counterfactual trend, or parallel or common trend for the treatment group that would occur in absence of treatment. The distance A'A represents a departure from the parallel trend in response to treatment, and would be our DD treatment effect or the value b₃ our estimator for δ.

The common trend assumption following Lechner, can be expressed in terms of potential outcomes:

E(Y₁⁰|D=1)-E(Y₀⁰|D=1) = α + δ⁰ + γ - α – γ = δ⁰ (8)

E(Y₁⁰|D=0)-E(Y₀⁰|D=0) = α + δ⁰ - α = δ⁰ (9)

i.e. the pre and post period differences in baseline outcomes is the same (δ⁰) regardless if individuals are assigned to the treatment group (D=1) or control group (D=0).

Nonlinear Models:

In a GLM framework, with a specific link function G(.) a DD framework can be expressed in terms of potential outcomes as follows:

E(Y_t¹|D) = G(α + tδ¹ + dγ) “potential outcome if treated” (10)

E(Y_t⁰|D) = G(α + tδ⁰ + dγ) “potential baseline outcome” (11)

DID can be estimated by regression on observed outcomes:

G(b₀ + b₁ d + b₂ t + b₃ d*t) (12)

Common Trend Assumption:

E(Y₁⁰|D=1)-E(Y₀⁰|D=1) = G(α + δ⁰ + γ) - G(α + γ) (13)

E(Y₁⁰|D=0)-E(Y₀⁰|D=0) = G(α + δ⁰ ) - G(α ) (14)

It turns out in a GLM framework, for the common trend assumption to hold, group specific differences must be zero or γ =0. The common trend assumption relies on differencing out specific terms of the unobservable potential outcome, or the individual specific effects we are trying to control for in the selection on unobservables scenario, but in a GLM scenario we have to assume that these effects are zero or absent. In essence, the attractive feature of DD models to control for unobservable effects is not a feature of DD models in a GLM scenario.  
References: 
The Estimation of Causal Effects by Difference-in-Difference Methods
By Michael Lechner Foundations and Trends in Econometrics
Vol. 4, No. 3 (2010) 165–224  

Program Evaluation and the
Difference-in-Difference Estimator
Course Notes
Education Policy and Program Evaluation
Vanderbilt University
October 4, 2008

Difference in Difference Models, Course Notes
ECON 47950: Methods for Inferring Causal Relationships in Economics
William N. Evans
University of Notre Dame
Spring 2008 

Propensity Score Matching Meets Difference-in-Differences

I recently have stumbled across a number of studies incorporating both difference-in-differences  (DD) and propensity score methods.  As discussed before, DD is a special case of fixed effects panel methods.  

In the World Bank's publication "Impact Evaluation in Practice" they give a nice summary of the power of DD in identification of causal effects:

"...we can conclude that many unobserved characteristics of individuals are also more or less constant over time. Consider, for example, a person's intelligence or such personality traits as motivation, optimism, self-discipline, or family health history...Interestingly, we are canceling out(or controling for) not only the effect of observed time invariant characteristics but also the effect of unobserved time invariant characteristics such as those mentioned above"

So with DD we can actually control for unobserved characteristics that we may not have data on or maybe couldn't measure appropriately or even quantify! That's powerful. In this framework we are controlling for unobservable characteristics that may be contributing to selection bias, we are achieving identification of treatment effects in a selection on unobservables context.

On the other hand, with propensity score matching, we are appealing to the conditional independence assumption, the idea that matched comparisons imply balance on observed covariates, which ‘recreates’ a situation similar to a randomized experiment  where all subjects are essentially the same except for the treatment(Thoemmes and Kim, 2011). Propensity score matching can identify treatment effects in a selection on observables context. 

But, what if we combine both approaches. The Impact Evaluation book has a section on mixed methods that gives a really good treatment of the power of using both PSM and DD:

"Matched difference-in-differences is one example of combining methods. As discussed previusly, simple propensity score matching cannot account for unobserved characteristics that might explain why a group chooses to enroll in a program and that might also affect outcomes. By contrast, matching combined with difference-in-differences at least takes care of any unobserved characteristics that are constant across time between the two groups"

Below are several papers that utilize the combination of DD and PSM:

Does Matching Overcome Lalonde’s Critique of Nonexperimental Estimators? Jeffrey Smith and Petra Todd. University of Maryland. 2003

Do Agricultural Land Preservation Programs Reduce Farmland Loss? Evidence from a Propensity Score Matching Estimator
Xiangping Liu and Lori Lynch January 2010

Measuring the Impact of Meat Packing and Processing Facilities in the Nonmetropolitan Midwest: A Difference- in-Differences Approach
Georgeanne M Artz, Peter Orazem, Daniel Otto
November 2005 Working Paper # 03003
Iowa State University

How Effective is Health Coaching in Reducing Health Services Expenditures?
Yvonne Jonk, PhD,* Karen Lawson, MD,w Heidi O’Connor, MS,z Kirsten S. Riise, PhD,y David Eisenberg, MD,8z Bryan Dowd, PhD,z and Mary J. Kreitzer, PhD, RN, FAANw
Medical Care 􏰃 Volume 53, Number 2, February 2015

References:

Impact Evaluation in Practice
Paul J. Gertler Sebastian Martinez, Patrick Premand,
Laura B. Rawlings and Christel M. J. Vermeersch
Default Book Series.December 2010

Interaction Models

Given a model of the form:

y= β₀ + β₁ X+ β₂Z  + β₃ XZ+ e

the relationship between X and Y is conditional on Z. The interaction term represents the effect of X on Y  conditional on the value of Z.

In ‘Understanding Interaction Models: Improving Empirical Analysis’ by Brambor, Clark, and Golder the following  schematic is presented:

 
As the schematic shows, β₂ represents the difference in intercepts between the two regression lines.

Notes:

Marginal Effect of X on Y: ∂Y/ ∂X = β₁  + β₃ Z

β₁ = effect of X on Y when Z =0

If XY is significant, that implies that the relationship between X and Y differs significantly between classes or values of Z.

It is possible that the effect of X on Y is significant for some values of Z even if the interaction term is not, hence you cannot base the inclusion of XZ in the model on the significance of the interaction term (Bramber et al, 2005).

In determining significance, the basic regression output typically does not provide sufficient information  and modifications are required (Bramber et al, 2005).

Kmenta (1971) provides the following comments regarding the significance of interactions and constitutive terms:

“When there are interaction terms in the equation, then any given explanatory variable may be represented not by one but several regressors. The hypothesis that this variable does not influence Y means that the coefficients of all regressors involving this variable are jointly zero”

As a result, the significance of X and the XZ term is given by the following F-test:

F = [ (R²₂ – R²₁) / (k₂ – k₁ )] / [(1-R²₂) / (N- k₂ -1)

K_n = # of variables in each model respectively (model including and excluding the interaction term and interaction variable)

R²_n = R-square for each respective model

N = total observations

The standard error of β₁  + β₃ Z = sqrt(V(β₁)  + Z² V(β₃ ) + 2 Z COV(β₁ ,β₃))

Constructing Odds Ratios from Logistic Models: e ^{β1  + β3 Z}

References:

Understanding Interaction Models: Improving Empirical Analyses. Thomas Bramber, William Roberts Clark, Matt Golder. Political Analysis (2006) 14:63-82

Elements of Econometrics. Jan Kmenta. Macmillan (1971)