Randomized Encouragement: When noncompliance may be a feature and not a bug

Many times in a randomized controlled trial (RCT) issues related to non-compliance arise. Subjects assigned to the treatment fail to comply, while in other cases subjects that were supposed to be in the control group actually receive treatment. Other times we may have a new intervention (maybe it is a mobile app or some kind of product, service, or employer or government benefit) that law, contract, or nature implies that it can be accessed by everyone in our population of interest. We know that if we let nature take its course, users, adopters, or engagers are very likely going to be a self selected group that is different from others in a number of important ways. In a situation like this it could be very hard to know if observed outcomes from the new intervention are related to the treatment itself, or explained by other factors related to characteristics of those who choose to engage.

In a 2008 article in the American Journal of Public Health, alternatives to randomized controlled trials are discussed, and for situations like this the authors discuss randomized encouragement:

 "participants may be randomly assigned to an opportunity or an encouragement to receive a specific treatment, but allowed to choose whether to receive the treatment."

In this scenario, less than full compliance is the norm, a feature and not a bug. The idea is to roll out access in conjunction with randomized encouragement. A randomized nudge.

For example, in Developing a Digital Marketplace for Family Planning: Pilot Randomized Encouragement Trial (Green, et. al;  2018) randomized encouragement was used to study the impact of a digital health intervention related to family planning:

“women with an unmet need for family planning in Western Kenya were randomized to receive an encouragement to try an automated investigational digital health intervention that promoted the uptake of family planning”

If you have a user base or population already using a mobile app you could randomize encouragement to utilize new features through the app. In other instances, you could randomize encouragement to use a new product, feature, or treatment through text messaging. Traditional ways this has been done is through mailers or phone calls.

While treatment assignment or encouragement is random, non-compliance or the choice to engage or not engage is not! How exactly do we analyze results from a randomized encouragement trial in a way that allows us to infer causal effects?  While common approaches include intent-to-treat (ITT) or maybe even per-protocol analysis, treatment effects for a randomized encouragement trial can also be estimated based on complier average causal effects or CACE.

CACEs compare outcomes for individuals in the treatment group who complied with treatment (engaged as a result of encouragement) with individuals in the control group who would have complied if given the opportunity to do so.  This is key. If you think this sounds a lot like local average treatment effects in an instrumental variables framework this is exactly what we are talking about.

Angrist and Pishke (2015) discuss how instrumental variables can be used in the context of a randomized controlled trial (RCT) with non-compliance issues:

 "Instrumental variable methods allow us to capture the causal effect of treatment on the treated in spite of the nonrandom compliance decisions made by participants in experiments....Use of randomly assigned intent to treat as an instrumental variable for treatment delivered eliminates this source of selection bias." 

Instrumental varaible analysis gives us an estimation of local average treatment effects (LATE), which are the same as CACE. In simplest terms, LATE is the average treatment effect for the sub-population of compliers in a RCT. Or, the compliers or engagers in a randomized encouragement design.

There are obviously some assumptions involved and more technical details. Please see the references and other links below to read more about the mechanics, assumptions, and details involved as well as some toy examples.

References:

Mastering 'Metrics: The Path from Cause to Effect Joshua D. Angrist and Jörn-Steffen Pischke. 2015.

Connell A. M. (2009). Employing complier average causal effect analytic methods to examine effects of randomized encouragement trials. The American journal of drug and alcohol abuse, 35(4), 253–259. doi:10.1080/00952990903005882

Green EP, Augustine A, Naanyu V, Hess AK, Kiwinda L
Developing a Digital Marketplace for Family Planning: Pilot Randomized Encouragement Trial
J Med Internet Res 2018;20(7):e10756

Stephen G. West, Naihua Duan, Willo Pequegnat, Paul Gaist, Don C. Des Jarlais, David Holtgrave, José Szapocznik, Martin Fishbein, Bruce Rapkin, Michael Clatts, and Patricia Dolan Mullen, 2008:
Alternatives to the Randomized Controlled Trial
American Journal of Public Health 98, 1359_1366, https://doi.org/10.2105/AJPH.2007.124446

See also: 

Intent to Treat, Instrumental Variables and LATE Made Simple(er) 

Instrumental Variables and LATE 

Instrumental Variables vs. Intent to Treat 

Instrumental Explanations of Instrumental Variables

A Toy Instrumental Variable Application

Other posts on instrumental variables...

Intent to Treat, Instrumental Variables and LATE Made Simple(er)

Many times in a randomized controlled trial (RCT) issues related to non-compliance arise. Subjects assigned to the treatment fail to comply, while in other cases subjects that were supposed to be in the control group actually receive treatment. One way to deal with non-compliance is through an intent-to-treat framework (ITT)

Gupdta describes ITT:

"ITT analysis includes every subject who is randomized according to randomized treatment assignment. It ignores noncompliance, protocol deviations, withdrawal, and anything that happens after randomization. ITT analysis is usually described as “once randomized, always analyzed."

In Mastering Metrics, Angrist and Pischke describe intent-to-treat analysis:

"In randomized trials with imperfect compliance, when treatment assignment differs from treatment delivered, effects of random assignment...are called intention-to-treat (ITT) effects. An ITT analysis captures the causal effect of being assigned to treatment."

While treatment assignment is random, non-compliance is not! Therefore if instead of using intent to treat comparisons we compared those actually treated to those untreated (sometimes termed 'as treated' analysis) we would get biased results. When there is non-compliance, there is the likelihood that a relationship exists between potential outcomes and the actual treatment received. While the ITT approach gives an unbiased causal estimate of the treatment effect, it is often a diluted effect because of non-compliance issues and can provide an underestimate of the true effect (Angrist, 2006).

Angrist and Pishke discuss how instrumental variables can be used in the context of a RCT with non-compliance issues:

 "Instrumental variable methods allow us to capture the causal effect of treatment on the treated in spite of the nonrandom compliance decisions made by participants in experiments....Use of randomly assigned intent to treat as an instrumental variable for treatment delivered eliminates this source of selection bias." 

The purpose of this post is to build intuition related to how an instrumental variable (IV) approach differs from ITT, and how it is not biased by selection related to non-compliance issues in the same way that an 'as treated' analysis would be.

My goal is to demonstrate with a rather simple data set how IVs tease out the biases from non-compliance and give us only the impact of treatment on the compliers also known as the local average treatment effect (LATE).

A great example of IV and ITT applied to health care can be found in Finkelstein et. al. (2013 & 2014) - See The Oregon Medicaid Experiment, Applied Econometics, and Causal Inference.

For another post walking through the basic mechanics of instrumental variables (IV) estimation using a toy data set see: A Toy IV Application.

Key Assumptions

Depending on how you frame it there are about 5 key things (assumptions if we want to call them that) we need to think about when leveraging instrumental variables - in humble language:

1) SUTVA - you can look that up but basically it means no interactions or spillovers between the treatments and controls - my getting treated does not make a control case have a better or worse outcome as a result

2) Random Assignment - that is the whole context of the discussion above - the instrument (Z) or treatment assignment must be random

3) The Exclusion Restriction - Treatment assignment impacts outcome only through the treatment itself. It is the treatment that impacts the outcome. There is nothing about being in the randomly assigned treatment group that would cause your outcome to be higher or lower in and of itself, other than actually receiving the treatment.  Treatment assignment is ignorable. This is often represented as: Z -> D -> Y where Z is the instrument or random assignment, D is an indicator for actually receiving the treatment, and Y is the outcome.

4) Non-zero causal effect of Z on D: Being assigned to the treatment group is highly correlated with actually receiving the treatment i.e. when Z =1 then D is usually 1 as well. (if these were perfectly correlated that would imply perfect compliance)

5) Monotonicity - We'll just call this an assumption of 'no-defiers.' It means that there are no cases that always do the opposite of what their treatment assignment indicates, i.e. if Z = 1 then D = 0 AND if Z =0 then D is always 1. Stated differently  we can't have cases where there are those that always get the treatment when assigned to the control group and never receive treatment when assigned to the treatment group.

Types of Non-Compliance

Given these assumptions, with monotonicity we end up with three different groups of people in our study:

Never Takers: those that refuse treatment regardless of treatment/control assignment.

Always Takers: those that get the treatment even if they are assigned to the control group.

Compliers: those that comply or receive treatment if assigned to a treatment group but do not receive treatment when assigned to control group.

The compliers are characterized as participants that receive treatment only as a result of random assignment. The estimated treatment effect for these folks is often very desirable and in an IV framework can give us an unbiased causal estimate of the treatment effect.  But how does this work?

Discussion

I have to first recommend a great post over at egap.org titled '10 Things to Know About Local Average Treatment Effects.' Most of my post is based on those well thought out examples.

Just to level set, the context of this discussion going forward is a RCT with the outcome measured as Y, and treatment assignment being used as the instrument Z. (this can be extended to apply to other scenarios using other types of instruments). Actual receipt of treatment, or treatment status, is indicated by D with D=1 indicating a receipt of treatment. So an ITT analysis would simply be a comparison of outcomes for folks randomly assigned to treatment (Z = 1) vs those that were controls (Z = 0) regardless of compliance or non-compliance (determined by D). An 'as treated' analysis would be a comparison of everyone that received the treatment (D = 1) vs. those that did not (D=0) regardless of randomization. This is a biased analysis. The IV or local average treatment effect (LATE) estimate is the difference in outcomes for compliers.

Going back to the original article by Angrist (1996), it discusses IVs, LATEs and the types of noncompliance as they relate to the assumptions we previously discussed. In that article they explain that the treatment status (D) of the always takers and never takers is invariant (uncorrelated) to random assignment Z. No matter what Z is, they are going to do what they are going to do.  But, we also know that Z  (by definition of compliance and assumption 4) is correlated with actual treatment assignment D for the compliers.

Lets consider a RCT with one sided non-compliance. In this case the controls are not able to receive the treatment by nature of the design. So there are no 'always takers' in this discussion. Below is a table summarizing a scenario like this with 100 people randomly assigned to treatment (Z = 1) and 100 controls (Z = 0). (This can be extended to include always takers and the post I mentioned before at egap.org will walk through that scenario)

Z = 1	Z = 0
Treatment	Control
Never Taker	Never Taker
N = 20	N = 20
D = 0	D = 0
Y = 5	Y = 5
Complier	Complier
N =80	N =80
D = 1	D = 0
Y = 25	Y = 20

For story telling purposes, let's assume the 'treatment' is a weight loss program. We've got some really unmotivated folks (never takers) in both the treatment and control group that just don't comply with the treatment. Let's say on average they all end up losing 5 pounds (Y = 5) regardless of the group they are in. On the other hand, we have more conscientious folks that if randomly assigned to treatment they will participate. But they are motivated and healthy. Even in absence of treatment their potential outcomes (weight loss) are pretty favorable. They are bound to lose 20 pounds even in absence of treatment.

As discussed before, we  can see how when there is non-compliance, there is the likelihood that a relationship exists between potential outcomes and the actual treatment received.

If we ignore treatment assignment, and just compare the average weight lost (y) for those that received treatment to all of those that did not we could run the following regression:

Y = β0 + β1 D + e      

with β1 = 10 (see the R code  that generates this data and these results)

We could calculate this by hand as: 25 - [(2/3)*20 + (1/3)*5)] = 10

We know that non-compliance biases this estimate.

The ITT estimate can be estimated as:

Y = β0 + β1 Z + e  

with β1 =  4

We can see from the data this is simply the difference in means between the treatment and control group: [.2*5 + .8*25] - [.2*5 + .8*20] = 21-17 = 4

We know from the discussion above and can see from the data that this is greatly diluted by noncompliance. But because of randomization this is an unbiased estimate.

Finally, the IV or local average treatment effect (LATE) estimate is the difference in outcomes for compliers.

Because our example above is contrived, the outcomes for the compliers is made explicit in the table above. If you know exactly who the compliers are the math would be straight forward:

LATE = 25 - 20 = 5

You can also get LATEs by dividing the ITT effect by the share of compliers:

4/.8 = 5

In a previous post, I've described how an IV estimate teases out only that variation in our treatment D that is unrelated to selection bias and relates it to Y giving us an estimate for the treatment effect of D that is less biased.

We can view this through the lens of a 2SLS modeling strategy:

Stage 1: Regress D on Z to get D*

D* = β0 + β1 Z + e

β1 only picks up the variation in Z that is related to D (i.e. quasi-experimental variation) and leaves all of the variation in D  related to non-compliance and selection in the residual term.  You can think of this as working like a filtering process.

Stage 2: Regress Y on D*

Y = β0 +βIV D* + e  

The second stage relates changes in Z (quasi-experimental variation) to changes in our target Y.

We can see (from the R code below) that our estimate βIV   = 5.

We can also get the same result (and correct standard errors) by using the ivreg function from the AER package in R:

summary(ivreg(y ~ D | Z,data =df))

R Code: https://gist.github.com/BioSciEconomist/a72fae6e01053fdb6d13c9a80d8e39f9

References:

Angrist, Joshua D., et al. “Identification of Causal Effects Using Instrumental Variables.” Journal of the American Statistical Association, vol. 91, no. 434, 1996, pp. 444–455. JSTOR, www.jstor.org/stable/2291629.

Angrist, J.D. J Exp Criminol (2006) 2: 23. https://doi.org/10.1007/s11292-005-5126-x

"The Oregon Experiment--Effects of Medicaid on Clinical Outcomes," by Katherine Baicker, et al. New England Journal of Medicine, 2013; 368:1713-1722. http://www.nejm.org/doi/full/10.1056/NEJMsa1212321

Medicaid Increases Emergency-Department Use: Evidence from Oregon's Health Insurance Experiment. Sarah L. Taubman,Heidi L. Allen, Bill J. Wright, Katherine Baicker, and Amy N. Finkelstein. Science 1246183Published online 2 January 2014 [DOI:10.1126/science.1246183]

Gupta, S. K. (2011). Intention-to-treat concept: A review. Perspectives in Clinical Research, 2(3), 109–112. http://doi.org/10.4103/2229-3485.83221

Instrumental Variables and LATE

Often in program evaluation we are interested in estimating the average treatment effect (ATE).  This is in theory the effect of treatment on a randomly selected person from the population. This can be estimated in the context of a randomized controlled trial (RCT) by a comparison of means between treated and untreated participants.

However, sometimes in a randomized experiment, some members selected for treatment may not actually receive treatment (if participation is voluntary, for example the Medicaid expansion in Oregon). In this case, sometimes researchers will compare differences in outcome between those selected for treatment vs those assigned to control groups. This analysis, as assigned or as randomized, is referred to as an intent-to-treat analysis (ITT). With perfect compliance, ITT = ATE.

As discussed previously, using treatment assignment as an instrumental variable  (IV) is another approach to estimating treatment effects. This is referred to as a local average treatment effect (LATE).

What is LATE and how does it give us an unbiased estimate of causal effects?

In simplest terms, LATE is the ATE for the sub-population of compliers in an RCT (or other natural experiment where an instrument is used).

In a randomized controlled trial you can characterize participants as follows: (see this reference from egap.org for a really great primer on this)

Never Takers: those that refuse treatment regardless of treatment/control assignment.

Always Takers: those that get the treatment even if they are assigned to the control group.

Defiers: Those that get the treatment when assigned to the control group and do not receive treatment when assigned to the treatment group. (these people violate an IV assumption referred to monotonicity)

Compliers: those that comply or receive treatment if assigned to a treatment group but do not recieve treatment when assigned to control group.

The outcome for never takers is the same regardless of treatment assignment and in effect cancel out in an IV analysis. As discussed by Angrist and Pishke in Mastering Metrics, the always takers are prime suspects for creating bias in non-compliance scenarios. These folks are typically the more motivated participants and likely would have higher potential outcomes or potentially have a greater benefit from treatment than other participants.  The compliers are characterized as participants that receive treatment only as a result of random assignment. The estimated treatment effect for these folks is often very desirable and in an IV framework can give us an unbiased causal estimate of the treatment effect. This is what is referred to as a local average treatment effect or LATE.

How do we estimate LATE with IVs?

One way LATE estimates are often described is as dividing the ITT effect by the share of compliers. This can also be done in a regression context. Let D be an indicator equal to 1 if treatment is received vs. 0, and let Z be our indicator (0,1) for the original randomization i.e. our instrumental variable. We first regress:

D = β₀ + β₁ Z + e  

This captures all of the variation in our treatment that is related to our instrument Z, or random assignment. This is 'quasi-experimental' variation. It is also an estimate of the rate of compliance. β₁ only picks up the variation in treatment D that is related to Z and leaves all of the variation and unobservable factors related to self selection (i.e. bias) in the residual term.  You can think of this as the filtering process.  We can represent this as: COV(D,Z)/V(Z). 

Then, to relate changes in Z to changes in our target Y we estimate β₂  or COV(Y,Z)/V(Z).

Y = β₀ +β₂ Z + e        

Our instrumental variable estimator then becomes:

β_IV = β₂ / β₁  or (Z’Z)^-1Z’Y / (Z’Z)^-1Z’D or COV(Y,Z)/COV(D,Z)  

The last term gives us the total proportion of ‘quasi-experimental variation’ in D related to Y. We can also view this through a 2SLS modeling strategy:

Stage 1: Regress D on Z to get D* or D = β₀ + β₁ Z + e 

Stage 2: Regress Y on D*  or  Y = β₀ +β_IV D* + e 

 As described in Mostly Harmless Econometrics, "Intuitively, conditional on covariates, 2SLS retains only the variation in s [D  in our example above] that is generated by quasi-experimental variation- that is generated by the instrument z"

Regardless of how you want to interpret β_IV, we can see that it teases out only that variation in  our treatment D that is unrelated to selection bias and relates it to Y giving us an estimate for the treatment effect of D that is less biased.

The causal path can be represented as:

Z →D→Y   

There are lots of other ways to think about how to interpret IVs. Ultimately they provide us with an estiamate of the LATE which can be interpreted as an average causal effect of treatment for those participants in a study whose enrollment status is determined completely by Z (the treatment assignment) i.e. the compliers and this is often a very relevant effect of interest. 

Marc Bellemare has some really good posts related to this see here, here, and here.

Instrumental Variables vs. Intent to Treat

 "ITT analysis includes every subject who is randomized according to randomized treatment assignment. It ignores noncompliance, protocol deviations, withdrawal, and anything that happens after randomization. ITT analysis is usually described as “once randomized, always analyzed”.

"ITT analysis avoids overoptimistic estimates of the efficacy of an intervention resulting from the removal of non-compliers by accepting that noncompliance and protocol deviations are likely to occur in actual clinical practice"  - Gupta, 2011

 In Mastering Metrics, Angrist and Pischke describe intent-to-treat analysis:

"In randomized trials with imperfect compliance, when treatment assignment differs from treatment delivered, effects of random assignment...are called intention-to-treat (ITT) effects. An ITT analysis captures the causal effect of being assigned to treatment."

While treatment assignment is random, non-compliance is not! Therefore if instead of using intent to treat comparisons we compared those actually treated to those untreated we would get biased results, because this is essentially making uncontrolled comparisons between treated and untreated subjects.

Angrist and Pishke describe how instrumental variables can be used in this context:

 “The instrumental variables (IV) method harnesses partial or incomplete random assignment, whether naturally occurring or generated by researchers"

 "Instrumental variable methods allow us to capture the causal effect of treatment on the treated in spite of the nonrandom compliance decisions made by participants in experiments....Use of randomly assigned intent to treat as an instrumental variable for treatment delivered eliminates this source of selection bias."

In  Intent-to-Treat vs. Non-Intent-to-Treat Analyses under Treatment Non-Adherence in Mental Health Randomized Trials there is a nice discussion of ITT and IV methods with applications related to clinical research.  Below is a nice treatment of IV in this context:

“Instrumental variables are assumed to emulate randomization variables, unrelated to unmeasured confounders influencing the outcome. In the case of randomized trials, the same randomized treatment assignment variable used in defining treatment groups in the ITT analysis is instead used as the instrumental variable in IV analyses. In particular, the instrumental variable is used to obtain for each patient a predicted probability of receiving the experimental treatment. Under the assumptions of the IV approach, these predicted probabilities of receipt of treatment are unrelated to unmeasured confounders in contrast to the vulnerability of the actually observed receipt of treatment to hidden bias. Therefore, these predicted treatment probabilities replace the observed receipt of treatment or treatment adherence in the AT model to yield an estimate of the as-received treatment effect protected against hidden bias when all of the IV assumptions hold.”

A great example of IV and ITT applied to health care can be found in Finkelstein et. al. (2013 & 2014) - See the Oregon Medicaid Experiment, Applied Econometics, and Causal Inference.

Over at the Incidental Economist, there was a nice discussion of ITT in the context of medical research that does a good job of explaining the rationale as well as when departures from ITT make more sense (such as safety and non-inferiority trials).

See also:  
Instrumental Explanations of Instrumental Variables

A Toy IV Application

Other IV Related Posts

References: 

Mastering ’Metrics:
The Path from Cause to Effect
Joshua D. Angrist & Jörn-Steffen Pischke
2015

Gupta, S. K. (2011). Intention-to-treat concept: A review. Perspectives in Clinical Research, 2(3), 109–112. http://doi.org/10.4103/2229-3485.83221

Ten Have, T. R., Normand, S.-L. T., Marcus, S. M., Brown, C. H., Lavori, P., & Duan, N. (2008). Intent-to-Treat vs. Non-Intent-to-Treat Analyses under Treatment Non-Adherence in Mental Health Randomized Trials. Psychiatric Annals, 38(12), 772–783. http://doi.org/10.3928/00485713-20081201-10

"The Oregon Experiment--Effects of Medicaid on Clinical Outcomes," by Katherine Baicker, et al. New England Journal of Medicine, 2013; 368:1713-1722. http://www.nejm.org/doi/full/10.1056/NEJMsa1212321

Medicaid Increases Emergency-Department Use: Evidence from Oregon's Health Insurance Experiment. Sarah L. Taubman,Heidi L. Allen, Bill J. Wright, Katherine Baicker, and Amy N. Finkelstein. Science 1246183Published online 2 January 2014 [DOI:10.1126/science.1246183] 

Detry MA, Lewis RJ. The Intention-to-Treat PrincipleHow to Assess the True Effect of Choosing a Medical Treatment. JAMA. 2014;312(1):85-86. doi:10.1001/jama.2014.7523

Do Friends Let Friends Do IV...or is all of that unobserved heterogeneity and endogeneity all in your head?

A few weeks ago, there was a post that caught my attention at the 'Kids Prefer Cheese' blog titled "Friends don't let Friends do IV" which was very critical of instrumental variable techniques. Around that same time, Marc Bellemare posted a contrasting piece, titled "Friends do let Friends do IV".

For some reason, I've written a number of posts recently related to instrumental variables, discussing different intuitive approaches to understanding them, or connections with directed acyclic graphs (DAGs).   In the past, I have discussed them in the context of omitted variable bias and unobserved heterogeneity and endogeneity.

Now some colleagues have introduced me to a few papers authored by Quin that really question the validity of using instruments in this context. In the first paper, Resurgence of the Endogeneity-Backed Instrumental Variable Methods, Quin states:

“Essentially, the paranoia grows out of the fallacy that independent error terms exist prior to model specification and carry certain ‘structural’ interpretation similar to other economic variables…..In fact, it is practically impossible to validate the argument of endogeneity bias on the ground of correlation between a regressor and the error term in a multiple regression setting, especially when the model fit remains relatively low. Notice how much the basis of the IV treatment for ‘selection on the unobservables’ is weakened once 'e' is viewed as a model-derived compound of unspecified miscellaneous effects. In general, error terms of statistical models are derived from model specification. As such, they are unsuitable for any ‘structural’ interpretation, e.g. see Qin and Gilbert (2001)”

Quin goes deeper into this in a later working paper, Time to Demystify Endogeneity Bias.

From the abstract-

"This study exposes the flaw in defining endogeneity bias by correlation between an explanatory variable and the error term of a regression model. Through dissecting the links which have led to entanglement of measurement errors, simultaneity bias, omitted variable bias and self
 -selection bias, the flaw is revealed to stem from a Utopia mismatch of reality directly with single explanatory variable models."

The paper gets pretty heavy in details, despite promises to keep the math at a minimum. One of the central arguments they make about "endogeniety bias syndrome" is to point out an apparent misunderstanding or misinterpretation of error terms in multivariable vs single variable regression that is often used in applied work to set the stage for doing IV:

"Error terms or model residuals have been long perceived as sundry composites of what modellers are unable and/or uninterested to explain since Frisch’s time....Since cov(z,e)≠ 0 is single variable based, the contents of the error term have to be adequately ‘pure’, definitely not a mixture of sundry composites, to sustain its significant presence.  Indeed,  textbook  discussions  of  endogeneity  bias,  be  it  associated  with  SB (simultaneity bias), measurement errors, OVB(omitted variable bias) or SSB (self-selection bias), are all built on simple regression models. As soon as these models are extended to multiple ones, the correlation becomes mathematically intractable. In a multiple regression, all the explanatory variables are mathematically equal. Designation of one  as  the  causing  variable  of  interest  and  the  rest  as  control  variables  is  purely  from  the substantive  standpoint.  The  premise, cov(x,e)≠ 0, implies  not  only cov(z,e)≠ 0 for  the entire  set  of control  variables,  but  also  the  set  being  exhaustive.  Both  conditions  are  almost impossible to meet in practice."

Quin also has an applied paper related to wage elasticities where some of these ideas are put into context. See the references below.

References:

Duo Qin (2015). Resurgence of the Endogeneity-Backed Instrumental Variable Methods. Economics: The Open-Access, Open-Assessment E-Journal, 9 (2015-7): 1—35. http://dx.doi.org/10.5018/economics-ejournal.ja.2015-7 

QIN, D. (2015) “Time  to  Demystify  Endogeneity  Bias” SOAS
Department  of  Economics  Working
Paper Series, No. 192, The School of Oriental and African Studies
192 Time to Demystify Endogeneity Bias (pdf)

Qin, D., S. van Huellen and QC. Wang. (2014), “What Happens to Wage Elasticities When We  Strip  Playometrics?  Revisiting  Married  Women  Labour  Supply  Model”, SOAS Department  of  Economics  Working  Paper  Series,  No.  190,  The  School  of  Oriental  and
African Studies  https://www.soas.ac.uk/economics/research/workingpapers/file97784.pdf 

Directed Acyclical Graphs (DAGs) and Instrumental Variables

Previously I discussed several of the most useful descriptions of instrumental variables that I have encountered through various sources. I was recently reviewing some of Lawlor's work related to Mendelian instruments and realized this was the first place I have seen the explicit use of directed acyclical graphs to describe how instrumental variables work.





In describing the application of Mendelian instruments, Lawlor et al present instrumental variables with the aid of directed acyclic graphs. They describe an instrumental variable (Z) as depicted above in the following way, based on three major assumptions:

(1) Z is associated with the treatment or exposure of interest (X)
(2) Z is independent of the unobserved  confounding factors (U) that impact both X and the outcome of interest (Y).
(3) Z is independent of both the outcome of interest Y given X, and the unobservable factors U. (i.e. this is the ‘exclusion principle’ in that Z impacts Y only through X)

Our instrumental variable estimate, βIV is the ratio of E[Y|Z]/E[X|Z],  which can be estimated by two-stage least squares:

X* = β0 + β1 Z +  e
Y = β0 + βIV X* + e

The first regression gets only variation in our treatment or exposure of interest related to Z, and leaves all the variation related to U in the residual term. The second regression estimates βIV, and retains only the ‘quasi-experimental’ variation in X related to the instrument Z.

References:
Stat Med. 2008 Apr 15;27(8):1133-63. Mendelian randomization: using genes as instruments for making causal inferences in epidemiology. Lawlor DA, Harbord RM, Sterne JA, Timpson N, Davey Smith G. Link: http://www.ncbi.nlm.nih.gov/pubmed/17886233

Causal diagrams for empirical research
BY JUDEA PEARL. Biometrika (1995),82,4,pp.669-710

Instrumental Explanations of Instrumental Variables

I have recently discussed Marc Bellemare's 'Metrics Monday posts, but he's written many many more applied econometrics posts that are really really good. Another example is his post, Identifying Causal Relationships vs. Ruling Out All Other Possible Causes. The post is not about instrumental variables per say, but, in this post, he describes IVs in this way:

"As many readers of this blog know, disentangling causal relationships from mere correlations is the goal of modern science, social or otherwise, and though it is easy to test whether two variables x and y are correlated, it is much more difficult to determine whether x causes y. So while it is easy to test whether increases in the level of food prices are correlated with episodes of social unrest, it is much more difficult to determine whether food prices cause social unrest."
 
"In my work, I try to do so by conditioning food prices on natural disasters. To make a long story short, if you believe that natural disasters only affect social unrest through food prices, this ensures that if there is a relationship between food prices and social unrest, that relationship is cleaned out of whatever variation which is not purely due to the relationship flowing from food prices to social unrest. In other words, this ensures that the estimated relationship between the two variables is causal. This technique is known as instrumental variables estimation."

The idea of 'cleaning' out the bias or endogeneity etc. is consistent with how I tried to build intuition for IVs before  depicting an instrumental variable as being like a 'filter' that picks up only variation in the treatment (CAMP) unrelated to an omitted variable (INDEX) or selection bias.

"A very non-technical way to think about this is that we are taking Z and going through CAMP to get to Y, and bringing with us only those aspects of CAMP that are unrelated to INDEX.  Z is like a filter that picks up only the variation in CAMP (what we may refer to as ‘quasi-experimental variation) that we are interested in and filters out the noise from INDEX.  Z is technically related to Y only through CAMP."

Z →CAMP→Y   

 (you can read the full post for more context)

See also: http://econometricsense.blogspot.com/2013/06/unobserved-heterogeneity-and-endogeneity.html


Below are some more examples of discussions and descriptions of instrumental variables that have been the most beneficial to my understanding:

Kennedy: “The general idea behind this estimation procedure is that it takes the variation in the explanatory variable that matches up with variation in the instrument (and so is uncorrelated with the error), and uses only this variation to compute the slope estimate. This in effect circumvents the correlation between the error and the troublesome variable, and so avoids the asymptotic bias”

Mastering Metrics:“The instrumental variables (IV) method harnesses partial or incomplete random assignment, whether naturally occurring or generated by researchers….."

“The IV method uses these three assumptions to characterize a chain reaction leading from the instrument to student achievement. The first link in this causal chain-the first stage-connects randomly assigned offers with KIPP attendance, while the second link-the one we’re after-connects KIPP attendance with achievement.”

Dr. Andrew Gelman with comments from Hal Varian: How to think about instrumental variables when you get confused

“Suppose z is your instrument, T is your treatment, and y is your outcome. So the causal model is z -> T -> y……. when I get stuck, I find it extremely helpful to go back and see what I've learned from separately thinking about the correlation of z with T, and the correlation of z with y. Since that's ultimately what instrumental variables analysis is doing.”

Hal Varian:

"You have to assume that the only way that z affects Y is through the treatment, T. So the IV model is
T = az + e
y = bT + d

It follows that
E(y|z) = b E(T|z) + E(d|z)

Now if we
1) assume E(d|z) = 0
2) verify that E(T|z) != 0
we can solve for b by division. Of course, assumption 1 is untestable.

An extreme case is a purely randomized experiment, where e=0 and z is a coin flip."

References:

A Guide to Econometrics. Peter Kennedy.

Mastering 'Metrics. Joshua Angrist and Jörn-Steffen Pischke 

Mendelian Instruments (Applied Econometrics meets Bioinformatics)

Recently I defended the use of quasi-experimental methods in wellness studies, and a while back I sort of speculated that genomic data might be useful in a quasi-experimental setting-but wasn’t sure how: 

“If causality is the goal, then merge 'big data' from the gym app with biometrics and the SNP profiles and employ some quasi-expermental methodology to investigate causality.”

Then this morning at marginal revolution I ran across a link to a blog post that mentioned exploiting mendelian variation as instruments for a particular study related to alcohol consumption.

This piece gives a nice intro I think:

Stat Med. 2008 Apr 15;27(8):1133-63. Mendelian randomization: using genes as instruments for making causal inferences in epidemiology.

Lawlor DA1, Harbord RM, Sterne JA, Timpson N, Davey Smith G

Link: http://www.ncbi.nlm.nih.gov/pubmed/17886233

“Observational epidemiological studies suffer from many potential biases, from confounding and from reverse causation, and this limits their ability to robustly identify causal associations. Several high-profile situations exist in which randomized controlled trials of precisely the same intervention that has been examined in observational studies have produced markedly different findings. In other observational sciences, the use of instrumental variable (IV) approaches has been one approach to strengthening causal inferences in non-experimental situations. The use of germline genetic variants that proxy for environmentally modifiable exposures as instruments for these exposures is one form of IV analysis that can be implemented within observational epidemiological studies. The method has been referred to as 'Mendelian randomization', and can be considered as analogous to randomized controlled trials. This paper outlines Mendelian randomization, draws parallels with IV methods, provides examples of implementation of the approach and discusses limitations of the approach and some methods for dealing with these.”

See also: QTL analysis in R and A Toy Instrumental Variable Application

The Oregon Medicaid Experiment, Applied Econometrics, & Causal Inference

Recently, the findings of a paper published in Science that finds an increase in ER visits among patients benefiting from expanded medicaid in Oregon has been in the news. I like this work because it represents a great example of applied econometrics and causal inference in the field of healthcare econometrics:

http://www.npr.org/blogs/health/2014/01/02/259128081/medicaid-expansion-boosted-emergency-room-visits-in-oregon 

"The result, said Finkelstein, was that the groups of people with or without insurance were identical, "except for the fact that some have insurance and some don't. You've literally randomized the allocation of insurance coverage."

If you are not familiar with the context, the state of Oregon expanded medicaid coverage (pre PPACA) but only to randomly selected winners of a lottery. About half the winners did not apply for and utilize the expanded coverage, so the only TRUE RANDOM comparisons involve lottery winners to losers. So, as Finkelstein is quoted, it is literally a randomization of the allocation of insurance. This is a valid analysis under an 'intent-to-treat' framework - comparing winners to losers overall. However, you cannot compare those 50% or so lottery winners that took the new coverage to losers without coverage and appeal to randomization or claim that the groups are identical and comparable. (there could be huge issues related to selection bias) However, the authors used  instrumental variables which made it possible to get an estimate of 'local average treatment effects' comparing those winners that took the new coverage to statistically similar losers, who likely would have taken coverage if they would have been winners:

"We compare outcomes between the “treatment group” (those randomly selected in the lottery) and the “control group” (those not randomly selected)......Our intent-to-treat analysis, comparing the outcomes in the treatment and control groups, provides an estimate of the causal effect of winning the lottery (and being permitted to apply for OHP Standard)."

"Of greater interest may be the effect of Medicaid coverage itself. Not everyone selected by the lottery enrolled in Medicaid; some did not apply and some who applied were not eligible for coverage. To estimate the causal effect of Medicaid coverage, we use a standard instrumental-variable approach with lottery selection as an instrument for Medicaid coverage. This analysis uses the lottery’s random assignment to isolate the causal effect of Medicaid coverage."

So, what is the practical difference between intent-to-treat and the local average treatment effect (via instrumental variables) in the context of this research? The authors explain that very well:

"The intent-to-treat estimate may be a relevant parameter for gauging the effect of the ability to apply for Medicaid; the local-average-treatment-effect estimate is the relevant parameter for evaluating the causal effect of Medicaid for those actually covered."

Also as discussed in the supplementary appendix, both the ITT and IV estimates are based on linear probability models and comparison to marginal effects derived from logistic regression. (see Oregon Medicaid Experiment and Linear Probability Models)

You can find a good discussion about this experiment intent to treat, etc in the context of the NEJM paper in an EconTalk podcast with Jim Manzi and Russ Roberts this past year: http://www.econtalk.org/archives/2013/05/jim_manzi_on_th.html 

Also, a nice profile of MIT economist Amy Finkelstein in a related story from Bloomberg: "MIT Economist Seeks Facts in Health-Care Policy Debate."  http://www.bloomberg.com/news/2014-01-03/mit-economist-seeks-facts-in-health-care-policy-debate.html 


References: 

"The Oregon Experiment--Effects of Medicaid on Clinical Outcomes," by Katherine Baicker, et al. New England Journal of Medicine, 2013; 368:1713-1722. http://www.nejm.org/doi/full/10.1056/NEJMsa1212321

Medicaid Increases Emergency-Department Use: Evidence from Oregon's Health Insurance Experiment. Sarah L. Taubman,Heidi L. Allen, Bill J. Wright, Katherine Baicker, and Amy N. Finkelstein. Science 1246183Published online 2 January 2014 [DOI:10.1126/science.1246183]

A Toy Instrumental Variable Application

I have previously discussed instrumental variables (here and here )  from a somewhat technical standpoint, but now I’d like to present a very basic example with a toy data set that demonstrates how IV estimation works in practice. The data set below is fabricated for demonstration purposes. The idea is to develop intuition about the mechanics of IV estimators, so we won’t concern ourselves with getting the appropriate standard errors in this example. 

Suppose an institution has a summer camp designed to prepare high school students for their first year of college and we want to assess the impacts of the camp on 1^st year retention.  The most basic fixed effect regression model to assess the impact of ‘CAMP’ might be specified as follows:

Y =β₀ + β₁ CAMP + β₂X + e            (1)

Where y = first year retention (see here  and here for a thorough and apologetic discussion of linear probability models)

CAMP = an indicator for camp attendance 

X = a vector of controls

Let’s simplify the discussion and exclude controls from the analysis for now. That leaves us with:

Y =β₀ + β₁ CAMP + e        (2)

The causal effect of interest or the treatment effect of CAMP is our regression estimate β₁ in the regression above.  If we use the data above we get an estimate of the treatment effect β₁ = .68 (i.e. CAMP attendance is associated with a 68% higher level of retention compared to students that don’t attend.) But, what if CAMP attendance is voluntary? If attendance is voluntary, then it could be that students that choose to attend also have a high propensity to succeed due to unmeasured factors (social capital, innate ability, ambition, etc.)  If this is the case, our observed estimate for β₁ could overstate the actual impact of CAMP on retention.  If we knew about a variable that captures the omitted factors that may be related to both the choice of attending the CAMP and having a greater likelihood of retaining (like social capital, innate ability, ambition, etc.) let’s just call it INDEX, we would include it and estimate the following:

Y =β₀ + β₁ CAMP +  β₂ INDEX + e                                (3)

We would get the following estimate  β₁ =0.3636  which would be closer to the true effect of CAMP. So, omitted variable bias in equation (2) is causing us to overestimate the effect of CAMP.  One way to characterize the selection bias problem is through the potential outcomes framework that I have discussed before, but this time lets characterize this problem in terms of the regression specification above. By omitting INDEX, information about INDEX is getting sucked up into the error term. When this happens, to the extent that INDEX is correlated with CAMP, CAMP becomes correlated with the error term ‘e.’  This correlation with the error term is a violation of the classical regression assumptions and leads to biased estimates of β₁, which we notice with the higher value (.68) that we get above when we omit INDEX. (For more technical terminology than ‘getting sucked up into the error term’ see my discussion about unobserved heterogeneity and endogeneity). 

So the question becomes, how do we tease out the true effects of CAMP, when we have this omitted variable INDEX that we can’t possibly measure that is biasing our estimate? Techniques using what are referred to as instrumental variables will help us do this. 

Let’s suppose we find some variable we hadn’t thought of called Z. Suppose that Z tends to be correlated with our variable of interest CAMP. For the most part, where Z = 1, CAMP = 1.  But we also notice (or argue) that Z tends to be unrelated to all of those omitted factors like innate ability and ambition that comprise the variable INDEX that we wish we had. The technique of instrumental variables looks at changes in a variable like Z, and relates them to changes in our variable of interest CAMP, and then relates those changes to the outcome of interest, retention.  Since Z is unrelated to INDEX, then those changes in CAMP that are related to Z are likely to be less correlated with INDEX (and hence less correlated with the error term ‘e’). A very non-technical way to think about this is that we are taking Z and going through CAMP to get to Y, and bringing with us only those aspects of CAMP that are unrelated to INDEX.  Z is like a filter that picks up only the variation in CAMP (what we may refer to as ‘quasi-experimental variation) that we are interested in and filters out the noise from INDEX.  Z is technically related to Y only through CAMP.

Z →CAMP→Y     (4)

If we can do this, then our estimate of the effects of  CAMP on Y will be unbiased by the omitted effects of INDEX.  So how do we do this in practice?

We can do this through a series of regressions.  To relate changes in Z to changes in CAMP  we estimate:

CAMP = β₀ + β₁ Z + e                (5)

Notice in (5), β₁ only picks up the variation in Z that is related to CAMP and leaves all of the variation in CAMP related to INDEX in the residual term.  You can think of this as the filtering process. Then, to relate changes in Z to changes in our target Y we estimate:

Y = β₀ +β₂ Z + e          (6)

Our instrumental variable estimator then becomes:

β_IV = β₂ / β₁  or (Z’Z)^-1Z’Y / (Z’Z)^-1Z’CAMP or COV(Y,Z)/COV(CAMP,Z)   (7)

The last term in (7) indicates that β_IV represents the proportion of total variation in CAMP that is related to our ‘instrument’ Z that is also related to Y.  Or, the total proportion of variation in CAMP unrelated to INDEX that is related to Y. Or, the total proportion of ‘quasi-experimental variation’ in CAMP related to Y.  Regardless of how you want to interpret β_IV, we can see that it teases out only that variation in CAMP that is unrelated to INDEX and relates it to Y giving us an estimate for the treatment effect of CAMP that is less biased than the standard regression like (2). In fact if we compute β_IV as in (7), we get β_IV = .3898. Notice this is much closer to what we think is the true estimate of β₁ that we would get from regression (3) if we had information about INDEX and could include it in the model specification.

Application in Higher Education

In their paper 'Using Instrumental Variables to Account for Selection Effects in Research on First Year Programs' Pike, Hansen and Lin account for self- selection using instrumental variables to get an unbiased measure the impact of first year programs. As I discussed this paper previously, in a normal multivariable regression specification, after including various controls, they find  a positive significant relationship between first year programs and student success (measured by GPA). However, by including the instruments in the regression (correcting for selection bias) this relationship goes away. In the paper they state:

"If, as the results of this study suggest, traditional evaluation methods can overstate (either positively or negatively) the magnitude of program effects in the face of self-selection, then evaluation research may be providing decision makers with inaccurate information. In addition to providing an incomplete accounting for external audiences, inaccurate information about program effectiveness can lead to the misallocation of scarce institutional resources."

Addendum: Estimating IV via 2SLS

The example above derives  β_IV  as discussed in a previous post. But, you can also get β_IV by substitution via two stage least squares (as discussed here):

CAMP = β₀ + β₁ Z + e                (5)

RET =  β₀ + β_IV CAMP^est + e    (8)

As discussed above, the first regression gets only the variation in CAMP related to Z, and leaves all of the variation in CAMP related to INDEX in the residual term. As Angrist and Pischke state, the second regression estimates β_IV and retains only the quasi-experimental variation in CAMP generated by the instrument Z. As discussed in their book Mostly Harmless Econometrics, most IV estimates are derived using packages like SAS, STATA, or R vs. explicit implementation of the methods illustrated above. Caution should be used to derive the correct standard errors, which are not the ones you will get in the intermediate results from any of the regressions depicted above. 

References: 

Angrist and Pischke, Mostly Harmless Econometrics, 2009 

Identification of Causal Effects Using Instrumental Variables

Joshua D. Angrist, Guido W. Imbens and Donald B. Rubin

Journal of the American Statistical Association , Vol. 91, No. 434 (Jun., 1996), pp. 444-455

Using Instrumental Variables to Account for Selection Effects in Research on First-Year Programs
Gary R. Pike, Michele J. Hansen and Ching-Hui Lin
Research in Higher Education
Volume 52, Number 2, 194-214

R Code for the Examples Above:

#   ------------------------------------------------------------------
#  | PROGRAM NAME: R_INSTRUMENTAL_VAR
#  | DATE:6/17/13 
#  | CREATED BY: MATT BOGARD 
#  | PROJECT FILE: P:\BLOG\STATISTICS             
#  |----------------------------------------------------------------
#  | PURPOSE: BASIC EXAMPLE OF IV
#  |
#  | 
#  |------------------------------------------------------------------
 
setwd('P:\\BLOG\\STATISTICS')
 
CAMP <- read.table("mplan.txt", header =TRUE)
 
 
summary(lm(CAMP$RET~CAMP$CAMP + CAMP$INDEX)) # true B1
summary(lm(CAMP$RET~CAMP$CAMP))  #  biased OLS estimate for B1
 
cor(CAMP$CAMP,CAMP$Z) # is the treatment correlated with the instrument?
cor(CAMP$INDEX,CAMP$Z) # is the instrument correlated with the omitted variable
 
cov(CAMP$RET,CAMP$Z)/var(CAMP$Z) # y = bZ
cov(CAMP$CAMP,CAMP$Z)/var(CAMP$Z) # x = bZ
 
# empirical estiamte of B1(IV)
 
(cov(CAMP$RET,CAMP$Z)/var(CAMP$Z))/(cov(CAMP$CAMP,CAMP$Z)/var(CAMP$Z)) # B1(IV)
 
# or
 
(cov(CAMP$RET,CAMP$Z))/(cov(CAMP$CAMP,CAMP$Z)) 
 
# two stage regression with IV substitution
 
# x^ = b1Z
 
CAMP_IV <- predict(lm(CAMP$CAMP~CAMP$Z)) # produce a vector of estimates for x
 
# y = b1 x^
 
lm(CAMP$RET~CAMP_IV)

Created by Pretty R at inside-R.org