Fat Tails, Kurtosis, and Risk

When I think of 'fat tails' or 'heavy tails' I typically think of situations that can be described by probability distributions with heavy mass in the tails. This might imply that the tails are 'fat' or 'thicker' than other situations with less mass in the tails. (for instance, a normal distribution might be said to have thin tails while a distribution with more mass in the tails than the normal might be considered a 'fat tailed' distribution.)

Basically when I think of tails I think of 'extreme events', some occurrence with an excessive departure from what's expected on average. So, if there is more mass in a the tail of a probability distribution (it is a fat or thick tailed distribution) that implies that extreme events will occur with greater probability.  So in application, if I am trying to model or assess the probability of some extreme event (like a huge loss on an investment) then I better get the distribution correct in terms of tail thickness.

See here for a very  nice explanation of fat tailed events from the Models and Agents blog: http://modelsagents.blogspot.com/2008/02/hit-by-fat-tail.html

Also here is a great podcast from the CME group related to tail hedging strategies: (May 29 2014) http://www.cmegroup.com/podcasts/#investorsAndConsultants

And here's a twist, what if I am trying to model the occurrence of multiple events simultaneously? (a huge loss in commodities and equities simultaneously, or losses on real estate in Colorado and Florida simultaneously). I would want to model a multivariate process that captures the correlation or dependence between multiple extreme events (in other words between the tails of their distributions). Copulas offer an approach to modeling tail dependence, and again, getting the distributions correct matters.

When I think of how do we assess or measure tail thickness in a data set, I think of kurtosis. Rick Wicklin recently had a nice piece discussing the interpretation of kurtosis and relating kurtosis to tail thickness.

"A data distribution with negative kurtosis is often broader, flatter, and has thinner tails than the normal distribution."

" A data distribution with positive kurtosis is often narrower at its peak and has fatter tails than the normal distribution."

The connection between kurtosis can be tricky and kurtosis  cannot be interpreted this way universally in all situations. Rick gives some good examples if you want to read more.  But this definition of kurtosis from his article seems keep us honest:

"kurtosis can be defined as "the location- and scale-free movement of probability mass from the shoulders of a distribution into its center and tails. " with the caveat - "the peaks and tails of a distribution contribute to the value of the kurtosis, but so do other features."

But Rick also had an earlier post on fat and long tailed distributions where he puts all of this into perspective in terms of the connection to modeling extreme events as well as a more rigorous discussion and definition of tails and what 'fat' or 'heavy' tailed means:

"Probability distribution functions that decay faster than an exponential are called thin-tailed distributions. The canonical example of a thin-tailed distribution is the normal distribution, whose PDF decreases like exp(-x2/2) for large values of |x|. A thin-tailed distribution does not have much mass in the tail, so it serves as a model for situations in which extreme events are unlikely to occur.

Probability distribution functions that decay slower than an exponential are called heavy-tailed distributions. The canonical example of a heavy-tailed distribution is the t distribution. The tails of many heavy-tailed distributions follow a power law (like |x|–α) for large values of |x|. A heavy-tailed distribution has substantial mass in the tail, so it serves as a model for situations in which extreme events occur somewhat frequently."

Also, somewhat related, Nassim Taleb, in his paper on the precautionary principle and GMOs (genetically modified organisms) discusses such concepts as ruin, harm,  fat tails and fragility, tail sensitivity to uncertaintly etc.  He uses very rigorous definitions of these terms and determines that there are certain things like GMOs that would require a non-naive application of the precautionary principle while other things like nuclear energy would not. (also tune into his discussion of this with Russ Roberts on Econtalk- more discussion on this actual application at my applied economics blog Economic Sense).

A Credibility Revolution in Wellness Program Analysis?

Last november there was a post on the Health Affairs blog related to the evaluation of wellness programs. Here are some tidbits:

"This blog post will consider the results of two compelling study designs — population-based wellness-sensitive medical event analysis, and randomized controlled trials (RCTs). Then it will look at the popular, although weaker, participant vs. non-participant study design."

"More often than not wellness studies simply compare participants to “matched” non-participants or compare a subset of participants (typically high-risk individuals) to themselves over time."

“Looking at how participants improve versus non-participants…ignores self-selection bias. Self-improvers are likely to be drawn to self-improvement programs, and self-improvers are more likely to improve.” Further, passive non-participants can be tracked all the way through the study since they cannot “drop out” from not participating, but dropouts from the participant group—whose results would presumably be unfavorable—are not counted and are considered lost to follow-up. So the study design is undermined by two major limitations, both of which would tend to overstate savings."

Does Wellness need a credibility revolution?

These criticisms are certainly valid, however, my thoughts are that panel methods, difference-in-difference and propensity score matching fit firmly in the Rubin Causal Model or potential outcomes framework for addressing issues related to selection bias. And what about examples of more robust quasi experimental approaches (like instrumental variables)?   These are all methods that are meant to deal specifically with the drawbacks of self comparisons and the issues mentioned above by the authors, and are at the heart of techniques related to the credibility revolution in econometrics.

A RCT is by far the most reliable way to identify treatment effects, but I know when it comes to applied work, RCT just isn't happening for a lot of obvious reasons. As Marc Bellemare might say, let the credibility revolution flow through you!

***this post was revised July 22, 2016, originally titled "Are Quasi-Experimental Designs Off the Table in Wellness Program Analysis"

Overconfident Confidence Intervals

In an interesting post, "Playing Dumb on Statistical Significance" there is a discussion relating to Naomi Oreskes January 4 NYT piece Playing Dumb on Climate Change. The dialogue centers around her reference to confidence intervals and possibly an overbearing burden of proof that researchers apply related to statistical significance.

From the article:

"Although the confidence interval is related to the pre-specified Type I error rate, alpha, and so a conventional alpha of 5% does lead to a coefficient of confidence of 95%, Oreskes has misstated the confidence interval to be a burden of proof consisting of a 95% posterior probability. The “relationship” is either true or not; the p-value or confidence interval provides a probability for the sample statistic, or one more extreme, on the assumption that the null hypothesis is correct. The 95% probability of confidence intervals derives from the long-term frequency that 95% of all confidence intervals, based upon samples of the same size, will contain the true parameter of interest."

As mentioned in the piece, Oreskes writing does have a Bayesian ring to it and this whole story and critique makes me think of Kennedy's chapter on "The Bayesian Approach" in his book "A Guide to Econometrics".  I believe that people often interpret frequentist based confidence intervals  from a bayesian perspective. If I understand any of this at all, and I admit my knowledge of Bayesian econometrics is limited, then I think I have been a guilty offender at times as well.  In the chapter it is even stated that Bayesians tout their methods because Bayesian thinking is actually how people really think and that is why they so often misinterpret frequentist confidence intervals.

In Bayesian analysis, a posterior probability distribution is produced (and a posterior probability interval) that 'chops off' 2.5% from each tail leaving an area or probability of 95%. From a Bayesian perspective, it is correct for the researcher to claim or believe that there is a 95% probability that the true value of the parameter they are estimating will fall within the interval. This is how many people interpret confidence intervals, which are quite different from Bayesian posterior probability intervals. An illustration is given from Kennedy:

"How do you think about an unknown parameter? When you are told that the interval between 2.6 and 2.7 is a 95% confidence interval, how do you think about this? Do you think, I am willing to bet $95 to your $5 that the true value of 'beta' lies in this interval [note this sounds a lot like Oreskes as if you read the article above]? Or do you think, if I were to estimate this interval over and over again using data with different error terms, then 95% of the time this interval will cover the true value of 'beta'"?

"Are you a Bayesian or a frequentist?"

Econ(ometrics) Talk

I just recently purchased Angrist and Pischke's "Mastering Metrics" (HT Marc Bellemare). And timely, today's EconTalk podcast featured Josh Angrist:

"Joshua Angrist of the Massachusetts Institute of Technology talks to EconTalk host Russ Roberts about the craft of econometrics--how to use economic thinking and statistical methods to make sense of data and uncover causation. Angrist argues that improvements in research design along with various econometric techniques have improved the credibility of measurement in a complex world. Roberts pushes back and the conversation concludes with a discussion of how to assess the reliability of findings in controversial public policy areas."

I've mentioned their previous book, Mostly Harmless Econometrics on this blog many times before and had some actual interaction with the authors via their related blog where I asked a regression/matching related question. I have described their book as an off-road backwoods survival manual for practitioners.
 
To say the least I am looking forward to the podcast and reading their latest book.

Some related Posts:

Angrist and Pischke on Linear Probabiity Models
Applied Econometrics
Quasi-Experimental Design Roundup
Analytics vs Causal Inference
The Oregon Experiment, Applied Econometrics, and Causal Inference
The Oregon Experiment and Linear Probability Models

Some related EconTalk podcasts that I highly recommend:

Leamer on the State of Econometrics
Manzi on the Oregon Medicaid Study
Manzi on Knowledge, Policy, and Uncontroled

A Cookbook Econometrics Analogy

Previously I wrote a post on applied econometrics motivated by a previous post made by Marc Bellemare and Dave Giles.

Since then, I've been reading Kennedy's chapter on applied econometrics in greater detail (I have a 6th edition copy) and I found the following interesting analogy. Typically cookbook analogies relate negatively to practitioners mind numbingly running regressions and applying tests etc. without strong appreciation for the underlying theory, but this is of a different flavor and to me gives a good impression of what 'doing econometrics' actually feels like:

From The Valavanis (1959, p.83) in Kennedy 6th Edition:

"Econometric theory is like an exquisitely balanced French recipe, spelling out precisely with how many turns to mix the sauce, how many carats of spice to add, and for how many milliseconds to bake the mixture at exactly 474 degrees of temperature. But when the statistical cook turns to raw materials, he finds that hearts of cactus fruit are unavailable, so he substitutes chunks of cantaloupe; where the recipe calls for vermicelli he used shredded wheat; and he substitutes green garment die for curry, ping-pong balls for turtles eggs, and for Chalifougnac vintage 1883, a can of turpentine."

It really gets uncomfortable when you are presenting at a seminar or conference or other audeince and someone that isn't elbow deep in the data challenges points out that your estimator isn't valid theoretically because you used 'turpentine' when the recipe (or econometric theory) calls for  Chalifougnac vintage 1883 or someone well-versed in theory but unaware of the social norms of applied econometrics tries to make you look incompetent by pointing out this 'mistake.'

Also gives me  That Modeling Feeling.

Applied Econometrics

I really enjoy Marc Bellemare's applied econ posts, but I really enjoy his econometric related posts (for instance a while back he wrote some really nice posts related  linear probability models   here and here).

Recently he wrote a piece entitled "In defense of the cookbook  approach to econometrics." At one point he states:

"The problem is that there is often a wide gap between the theory and practice of econometrics. This is especially so given that the practice of econometrics is often the result of social norms..."

He goes on to make the point that 'cookbook' classes should be viewed as a complement, not a supplement to theoretical econometrics classes.

It is the gap between theory and practice that has given me a ton of grief as a practitioner. The amount of grief has varied depending on if work was directed strictly toward problem solving or publication (i.e. bowing to the seemingly eccentric whims and wishes of a reviewer). You can spend hours in graduate school working through tons of theorems and proofs or have a mis-spent youth toughing your way through the literature and still find your self on what feels like another planet when it comes to actually doing econometrics. At every corner you will find a different challenge that seems at odds with everything you may have learned in your econometric theory course, textbook, or paper that everyone else cites in their work. As Peter Kennedy states in the applied econometrics chapter of his popular A Guide to Econometrcs, "econometrics is much easier without data".

He states further:

"In virtually every econometric analysis there is a gap, usually a vast gulf, between the problem at hand and the closest scenario to which standard econometric theory is applicable....the issue here is that in their econometric theory courses students are taught standard solutions to standard problems, but in practice there are no standard problems...Applied econometricians are continually faced with awkward compromises..."


The hard part for the recent graduate or new practitioner that has not had a good applied econometrics course or been exposed to the social norms and practices of applied work is figuring out how to compromise, or which sins are more forgivable or harmless than others. They may not no that sometimes you should Do Both and the difference between a garden of forking paths and sensitivity analysis.

As Angrist and Pischke state: "if applied econometrics were easy theorists would do it."


See also:

Culture War: Classical Statistics vs. Machine Learning

Ambitious vs. Ambiguous Modeling

Mostly Harmless Econometrics as an off-road backwoods survival manual for practitioners.

***updated and revised 6/22/2024

Implications of Maximum Likelihood Methods for Missing Data in Predictive Modeling Applications

I believe there are some key things to consider when we deal with situations where we have missing data when we are 1) estimating a model to be used for prediction (training) and 2) using the model to predict new cases (scoring) which also may have missing values for key predictor variables.  For purposes of this discussion, I'm thinking specifically about situations where one intends to use ML or FIML to estimate the parameters that define or train your model.  You must consider how to handle missing values in both the model training and scoring exercises. Also, there may be distinctions to consider in a purely predictive/machine learning application vs. causal inference.

If your goal is simply to estimate paramter values to make causal inferences i.e. evaluate treatment effects, then most likely you will only be concerned with imputation during the training or estimation stage. Again, in this post I am concerned with predictive modeling applications vs. causal inference . I will start with a short discussion of  maximum likelihood estimation. 

Standard Maximum Likelihood: 

Maximize L = Π f(y,x₁,…x_k;β)  

With standard ML, the likelihood function is optimized providing the values for β which define our regression model. (like Y = β₀ + β₁ x₁  + … β_k x_k  + e)

 As is the case in many modeling scenarios, with standard MLE, only complete cases are used to estimated the model. That is, for each 'row' or individual case, all values of 'x' and 'y' must be defined. If a single explanatory variable 'x' or the dependent variable 'y' have a missing value, then that individual/case/row is excluded from the data. This is referred to as listwise deletion. In many scenarios, this can be undesirable because for one thing, you are reducing the amount of information used to estimate you model. Paul Allison has a very informative discussion of this in a recent post at Statistical Horizons.

Full Information Maximum Likelihood (FIML): 

Maximize L = Π f(y,x₁,…x_k;β)  Π f(y,x₃,…x_k;β)  

Full information maximum likelihood is an estimation strategy that allows for us to get parameter estimates even in the presence of missing data.  The overall likelihood is the product of the likelihoods specified for all observations. If there are m observations with no missing values but n observations missing x₁ and x₂ we account for that by specifying the overall likelihood function as the product of two terms i.e. likelihood function is specified as a product of likelihoods for both complete and incomplete cases.  In the example above the second term in the product depicts a case where for individual ‘i’ there are missing values for the first 3 variables. The first term represents the likelihood for all other complete cases. The overall likelihood is then optimized providing the values for β which define our regression model.

Both ML and FIML are methods for estimating parameters; they are not imputation procedures per-say.  As Karen Grace Martin (Analysis Factor) aptly puts it  “This method does not impute any data, but rather uses each case's available data to compute maximum likelihood estimates.”

Predictive Modeling Applications

So if we have missing data, we could use FIML to obtain parameter estimates for a model, but what if we actually want to predict outcomes ‘y’ for some new data set (i.e. we want to 'score' a new data set using the model we just estimated). By assumption, if we are trying to predict ‘y’ we don’t have values for y in our data set.  We will attempt to take the model or parameter estimates we got from FIML and predict Y based on the estimated values of  our β’s and observed x’s. But what if in the new data we have missing x’s? Can’t we just use FIML to get our model and predictions? No. First we have already derived our model via FIML using our original or training data. Again, FIML is a model or parameter estimation procedure. To apply FIML in our new data set would imply 2 things:

1)   We want to estimate a new model on new data

2)  We have observed values for what we are trying to predict ‘y’ which by assumption we don’t have that in a prediction or scoring scenario! 

So there is no way to properly specify the likelihood to even implement FIML to estimate a new model in a new data set!

But, we don't want to estimate a new model in the first place. If we want to make new predictions based on our original model developed using FIML, we have to utilize some type of actual imputation procedure to derive values for missing x’s in the new 'scoring' data set.

 

References:

SAS Global Forum Paper 312-2012

Handling Missing Data by Maximum Likelihood

Paul D. Allison, Statistical Horizons, Haverford, PA, USA

Two Recommended Solutions for Missing Data: Multiple Imputation and Maximum Likelihood. Karen Gace-Martin. The Analysis Factor: http://www.theanalysisfactor.com/missing-data-two-recommended-solutions/ Accessed 8/14/14

Listwise Deletion: It's Not Evil. Paul Allison, Statistical Horizons. June 13,2014. http://www.statisticalhorizons.com/listwise-deletion-its-not-evil