Tuesday, December 29, 2015

Nonparametric Approaches to Multiple Comparisons

I have recently started reading "Applied Nonparametric Econometrics", and was thinking, when was the last time I even worked with basic non-parametric statistics?

For instance, in the courses I teach, I don't cover this, but some of the texts I reference cover some basics like the Mann Whitney Wilcoxon  (MWW) test (which can be thought of as a non-parametric equivalent to a two sample independent t-test) or the Kruskall-Wallis test (which is a non-parametric analogue to analysis of variance). These tests are often useful in situations that involve highly skewed, non-normal, or categorical ordered or  ranked data, or data from problematic or unknown distributions.  I kind of briefly reviewed some implementations in SAS, and particularly focused on the Kruskall-Wallis test, which has the following general null hypothesis:

Ho: All Populations Are Equal
Ha: All Populations Are Not Equal

If we reject Ho, we might conclude that there is a difference among populations, with one population or another providing a larger proportion of larger or smaller values for the variable of interest. If we could assume that the populations were of similar shape and symmetry, this *might* be interpreted as a test of differences in medians, but in general this is a test on differences in distributions and specifically ranks, similar to the MWW test. But if we do reject Ho, what next? In an analysis of variance context, if we reject the overall F-test on multiple means we can followup with pairwise comparisons to determine which means differ.  But at least in the older versions of SAS, there are no straightforward ways to do this kind of analysis in the non-parametric context. However, in the SAS Note (22620), one recommendation is to rank-transform the data and use the normal-theory methods in PROC GLM (Iman, 1982). See also Conover, W. J. & Iman, R. L. (1981) referenced below.

A good example of the application of GLM on ranked data can be found here: http://people.stat.sc.edu/Hitchcock/soil_KW_sasexample705.txt 

and a general overview of some non-parametric applications in SAS along these lines here.

You can also find a SAS macro with code and examples for post hoc tests here: http://www.alanelliott.com/kw/

I at first thought this was the macro by Juneau (in the references below and mentioned in the SAS note above) but it is something different, see the Elliot and Hynan reference below. From the abstract:

"The Kruskal-Wallis (KW) nonparametric analysis of variance is often used instead of a standard one-way ANOVA when data are from a suspected non-normal population. The KW omnibus procedure tests for some differences between groups, but provides no specific post hoc pair wise comparisons. This paper provides a SAS(®) macro implementation of a multiple comparison test based on significant Kruskal-Wallis results from the SAS NPAR1WAY procedure. The implementation is designed for up to 20 groups at a user-specified alpha significance level. A Monte-Carlo simulation compared this nonparametric procedure to commonly used parametric multiple comparison tests."

I found an application referencing this implementation here if interested.

According to the SAS note referenced above, SAS/STAT 12.1 will include some versions of some non-parametric post hoc tests. I'm also aware that there are several R packages that can do this as well, such as the dunn.test package.

I compared results from Elliot and Hynan's example code (example 1) and data to those from the adhoc GLM on ranks following Hitchcock and got similar results. I also got similar results using dunn.test in R:

# use same data as in www.alanelliott.com/kw
race <- c(1,1,1,1,1,2,2,2,2,2,3,3,3,3,3)
bmi <- c(32,30.1,27.6,26.2,28.2,26.4,23.1,23.5,24.6,24.3,24.9,25.3,23.8,22.1,23.4)
library(dunn.test) #load package
dunn.test(bmi,race, kw = TRUE, method ="bonferroni") # implement test with adjustments for multiple comparisons
Created by Pretty R at inside-R.org

Palomares-Rius JE, Castillo P, Montes-Borrego M, Navas-Cortés JA, Landa BB (2015) Soil Properties and Olive Cultivar Determine the Structure and Diversity of Plant-Parasitic Nematode Communities Infesting Olive Orchards Soils in Southern Spain. PLoS ONE 10(1): e0116890. doi:10.1371/journal.pone.0116890

Dunn, O.J. “Multiple comparisons using rank sums”.
Technometrics 6 (1964) pp. 241-252.

Conover, W. J. & Iman, R. L. (1981). "Rank transformations as a bridge between parametric and
nonparametric statistics". American Statistician 35 (3): 124–129. doi:10.2307/2683975

Elliott AC, Hynan LS. “A SAS Macro implementation of a Multiple Comparison post hoc test for a Kruskal-Wallis analysis,” Comp Meth Prog Bio, 102:75-80, 2011

Iman, R.L. (1982), "Some Aspects of the Rank Transform in Analysis of Variance Problems," Proceedings of the Seventh Annual SAS Users Group International Conference, 7, 676-680.

Juneau, P. (2004), "Simultaneous Nonparametric Inference in a One-Way Layout Using the SAS System," Proceedings of the PharmaSUG 2004 Annual Conference, Paper SP04.

Sunday, December 6, 2015

Do We Really Need Zero-Inflated Models?-Paul Allison

Paul Allison discusses zero inflated vs negative binomial models in a post I stumbled across recently. Also William Greene and Paul go back and forth on some technical distinctions and nuances (which may be quite important) in the comments.


"In all data sets that I've examined, the negative binomial model fits much better than a ZIP model, as evaluated by AIC or BIC statistics. And it's a much simpler model to estimate and interpret. So if the choice is between ZIP and negative binomial, I'd almost always choose the latter."

"But what about the zero-inflated negative binomial (ZINB) model? It's certainly possible that a ZINB model could fit better than a conventional negative binomial model regression model. But the latter is a special case of the former, so it's easy to do a likelihood ratio test to compare them (by taking twice the positive difference in the log-likelihoods). In my experience, the difference in fit is usually trivial..."

"So next time you're thinking about fitting a zero-inflated regression model, first consider whether a conventional negative binomial model might be good enough. Having a lot of zeros doesn't necessarily mean that you need a zero-inflated model."

Saturday, December 5, 2015

Do Friends Let Friends Do IV...or is all of that unobserved heterogeneity and endogeneity all in your head?

A few weeks ago, there was a post that caught my attention at the 'Kids Prefer Cheese' blog titled "Friends don't let Friends do IV" which was very critical of instrumental variable techniques. Around that same time, Marc Bellemare posted a contrasting piece, titled "Friends do let Friends do IV".

For some reason, I've written a number of posts recently related to instrumental variables, discussing different intuitive approaches to understanding them, or connections with directed acyclic graphs (DAGs).   In the past, I have discussed them in the context of omitted variable bias and unobserved heterogeneity and endogeneity.

Now some colleagues have introduced me to a few papers authored by Quin that really question the validity of using instruments in this context. In the first paper, Resurgence of the Endogeneity-Backed Instrumental Variable Methods, Quin states:

“Essentially, the paranoia grows out of the fallacy that independent error terms exist prior to model specification and carry certain ‘structural’ interpretation similar to other economic variables…..In fact, it is practically impossible to validate the argument of endogeneity bias on the ground of correlation between a regressor and the error term in a multiple regression setting, especially when the model fit remains relatively low. Notice how much the basis of the IV treatment for ‘selection on the unobservables’ is weakened once 'e' is viewed as a model-derived compound of unspecified miscellaneous effects. In general, error terms of statistical models are derived from model specification. As such, they are unsuitable for any ‘structural’ interpretation, e.g. see Qin and Gilbert (2001)”

Quin goes deeper into this in a later working paper, Time to Demystify Endogeneity Bias.

From the abstract-

"This study exposes the flaw in defining endogeneity bias by correlation between an explanatory variable and the error term of a regression model. Through dissecting the links which have led to entanglement of measurement errors, simultaneity bias, omitted variable bias and self
 -selection bias, the flaw is revealed to stem from a Utopia mismatch of reality directly with single explanatory variable models."

The paper gets pretty heavy in details, despite promises to keep the math at a minimum. One of the central arguments they make about "endogeniety bias syndrome" is to point out an apparent misunderstanding or misinterpretation of error terms in multivariable vs single variable regression that is often used in applied work to set the stage for doing IV:

"Error terms or model residuals have been long perceived as sundry composites of what modellers are unable and/or uninterested to explain since Frisch’s time....Since cov(z,e)≠ 0 is single variable based, the contents of the error term have to be adequately ‘pure’, definitely not a mixture of sundry composites, to sustain its significant presence.  Indeed,  textbook  discussions  of  endogeneity  bias,  be  it  associated  with  SB (simultaneity bias), measurement errors, OVB(omitted variable bias) or SSB (self-selection bias), are all built on simple regression models. As soon as these models are extended to multiple ones, the correlation becomes mathematically intractable. In a multiple regression, all the explanatory variables are mathematically equal. Designation of one  as  the  causing  variable  of  interest  and  the  rest  as  control  variables  is  purely  from  the substantive  standpoint.  The  premise, cov(x,e)≠ 0, implies  not  only cov(z,e)≠ 0 for  the entire  set  of control  variables,  but  also  the  set  being  exhaustive.  Both  conditions  are  almost impossible to meet in practice."

Quin also has an applied paper related to wage elasticities where some of these ideas are put into context. See the references below.


Duo Qin (2015). Resurgence of the Endogeneity-Backed Instrumental Variable Methods. Economics: The Open-Access, Open-Assessment E-Journal, 9 (2015-7): 1—35. http://dx.doi.org/10.5018/economics-ejournal.ja.2015-7 

QIN, D. (2015) “Time  to  Demystify  Endogeneity  Bias” SOAS
Department  of  Economics  Working
Paper Series, No. 192, The School of Oriental and African Studies
192 Time to Demystify Endogeneity Bias (pdf)

Qin, D., S. van Huellen and QC. Wang. (2014), “What Happens to Wage Elasticities When We  Strip  Playometrics?  Revisiting  Married  Women  Labour  Supply  Model”, SOAS Department  of  Economics  Working  Paper  Series,  No.  190,  The  School  of  Oriental  and
African Studies  https://www.soas.ac.uk/economics/research/workingpapers/file97784.pdf