Logit Models: R-square and the Percentage of Correct Predictions

In my post on logistic regression and maximum likelihood estimation, using measures of deviance (derived from the log-likelihood), I presented a formulation for a pseudo-R-square. The pseudo-R-square is OK, some people like it, but people often get wrapped up in the least squares framework, and start to talk and act like they are demonstrating explained sums of squared errors. This is not what is being measured. With maximum likelihood, we are not minimizing squared errors, but maximizing a specified likelihood function.   While  it is fashioned in the sense of a traditional R-square, with the pseudo R-square there are no sums of squares to compute 1-SSE/SST. The pseudo R-square simply attempts to capture realtive changes in the log likelihood given the model parameters. The various transformations (Cox and Snell, Nagelkerke) attempt to make it a little more useful.

Below are some remarks from Greene and Kennedy regarding the use of the pseuso R-square and assessing model performance in the framework of maximum likelihood and logistic regression.

‘But the maximum likelihood estimator, on which all the fit measures are

based, is not chosen so as to maximize a fitting criterion based on prediction of y as it is in the classical regression (which maximizes R2). It is chosen to maximize the joint density of the observed dependent variables. It remains an interesting question for research whether fitting y well or obtaining good parameter estimates is a preferable estimation criterion. Evidently, they need not be the same thing.’

 p. 686 Greene,  Econometric Analysis 5^th ed

There is no universally accepted goodness of fit measure (pseudo-Rsquare for probit, logit, or count data models). It is tempting to use the percentage of correct predictions as a measure of goodness of fit. This temptation should be resisted: a naïve predictor, for example  that every y=1, could do well on this criterion. A better measure along these lines is the sum of the fraction of zeros correctly predicted plus the fraction of ones correctly predicted, a number which should exceed unity if the prediction method is of value. ( p. 267 Kennedy, A Guide to Econometrics 5^th ed)

Kennedy makes a good point. Often times you will see results indicating that a logit model performs well because the percentage of correct predictions  is very high, maybe 90% or higher. But what if your population is dominated by cases belonging to class Y=1, and your model does a great job predicting those classes?  If all you are interested in involves predicting cases where Y=1, that may be an OK model. However, as Kennedy points out, if in your true population 9 out of every 10 observations belong to class Y=1, a naive predictor that classifies 10/10 cases as belonging to class 1 will also have a 90% percentage of correct predictions.  If you are interested in those cases belonging to class Y =0, the naive model will be wrong 100% of the time, and your actual model will be wrong more often than not.

More on R-square and Goodness of Fit for Discrete Choice Models

Thomas, Dawes, and Reznik. Using Predictive Modeling to Target Student Recruitment: Theory and Practice.  AIR Professional File 78 Winter 2001

 There is no universally accepted goodness of fit measure (pseudo-Rsquare for probit, logit, or count data models). It is tempting to use the percentage of correct predictions as a measure of goodness of fit. This temptation should be resisted: a naïve predictor, for example  that every y=1, could do well on this criterion. A better measure along these lines is the sum of the fraction of zeros correctly predicted plus the fraction of ones correctly predicted, a number which should exceed unity if the prediction method is of value. ( p. 267 Kennedy, A Guide to Econometrics 5^th ed)

Applied multiple regression/correlation analysis for the behavioral sciences 3rd Edition.

Jacob Cohen, Patricia Cohen - L. Erlbaum Associates (2003)

“Again, we caution that all these indicators are not goodness of fit indices in the sense of “proportion of variance accounted for”, in contrast to R² in OLS regression. This seems puzzling, perhaps, but the explanation is straight forward. Reflect for a moment on the OLS regression model, which assumes homoscedasticity-the same error variance for every value of the criterion. Given homoscedasticity, we are able to think of the total proportion of variance that is error variance in a universal sense, across the full range of Y. In contrast, in logistic regression, we have inherent heteroscedasticity, with a different error variance for each different value of the predicted score 'p'. "

Applied Logistic Regression. Second Edition. David W. Hosmer & Stanley Lemeshaw.

“The most common assumption is that 'e' follows a normal distribution with mean zero and some variance that is constant across all levels of the independent variable. It follows that the conditional distribution of the outcome variable given x will be normal with mean E(Y|x), and a variance that is constant. This is not the case with a dichotomous outcome variable…thus 'e'  has a distribution with mean zero and variance equal to π(x)[1-π(x)]. That is, the conditional distribution of the outcome variable follows a binomial distribution with probability given by the conditional mean, π(x).”

Data Mining and Input Transformations

In applied econometrics, there are often lots of reasons why we may or may not want to transform data. In the case of  time series data, we will take first differences to deal with non-stationarity.  To deal with serial correlation, we may resort to generalized least squares and adjust the data based on an estimate of the correlation of the error terms. In spatially weighted regression, the inputs are adjusted or weighted using a spatial weights matrix. In many of these cases, adjustments are made based on an attempt to explicitly model a theorized mathematical relationship between inputs and outputs.

In the SAS training 'Applied Analytics with SAS Enterprise Miner it is recommended to transform any of the training examples that may have skewed distributions, without regard to any explicit mathematical relationship between the explanatory and dependent variables.  The recommendation based on an attempt to eliminate leverage or skewness that may impact model performance. Recall, in regression the the goal is to use sample data to estimate the population conditional expectation function.  Since what we are estimating then is a conditional mean, it is subject to bias from outliers or the same sort of leverage that biases any mean from any skewed distribution.  (see also my post on data imputation and M-estimators).  A transformation that creates a more symmetric or less skewed distribution could lead to less biased predictions.

As explained in a great post at the stats blog for Children's Mercy Hospitals and Clinics (which may seem like an odd place for a stats blog, but a great resource from which I have often found great references):

"If you want to use a log transformation, you compute the logarithm of each data value and then analyze the resulting data. You may wish to transform the results back to the original scale of measurement.The logarithm function tends to squeeze together the larger values in your data set and stretches out the smaller values. This squeezing and stretching can correct one or more of the following problems with your data : 1. Skewed data  2. Outliers 3. Unequal variation"

Let's simulate a skewed data distribution and see how this works using R.

 The following R code will simulate skewed data, graph it,  and compute some basic diagnostics, as well as provide regression results for variable x2 and y:

# simulate skewed data 
 
x2 <- c(10,11,13,24,30,35,33,34,35,36,37,32,37,40,42,60,71,100,120, breaks =20)
y <- c(1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20)
 
hist(x2)        # plot histogram
summary(x2)     # summary stats
plot(x2,y)      # plot x2 and y
reg2 <- lm(y~x2)  # regress y on x2
abline(reg2)     # plot regression line over data
summary(reg2)    # regression output

Created by Pretty R at inside-R.org

 The  summary statistics give the following:

Min. 1st Qu.  Median    Mean 3rd Qu.    Max.
   10.0    28.5    35.0    41.0    40.5   120.0

It is easy to see that the mean is biased towards the tail of the distribution, and as expected is greater than the mean.

Regressing y on x2 gives an adjusted r-sqaare value of .4653.

Lets look at a log transformation of x2.

logx2 <- log(x2)    # log transform
hist(logx2)       # plot histogram of logx2
summary(logx2)      # produce summary stats
plot(logx2,y)       # scatterplot 
 
reg3 <- lm(y~logx2) # regress y on x2
abline(reg3)        # plot regresson line over data
summary(reg3)       # regression results

Created by Pretty R at inside-R.org

Taking a log transform of x2 gives a much more symmetrical distribution:

From the summary statistics, we see that the mean and median are almost identical, another indication that the transformation improved the symmetry of the distribution:

Min. 1st Qu.  Median    Mean 3rd Qu.    Max.
  2.303   3.345   3.555   3.521   3.701   4.787

From the regression output, we see that we are now able to explain more of the total variation in y after the transformation (squeezing the data together reduced the leverage exerted by the extreme observations in the sample and enabled a better fit- which would allow better predictions of y)

Adjusted R-square: .55

This is all without making any assumptions about the explicit mathematical relationship between x and y. In reality, could there be such an explicit relationship? Possibly. If theory leads you down that path then you should transform x regardless, not just to smooth out the distribution, but to incorporate the theory into your model.  Often times in data mining we are not sure of the relationships between the training examples and the dependent variable. But, if our goal is making good predictions or fitting the data well, we know that certain transformations can improve our results. As Greene says :

 "It remains an interesting question for research whether fitting y well or obtaining good parameter estimates is a preferable estimation criterion. Evidently, they need not be the same thing." (Econometric Analysis, 5th Edition)

Greene is referring to issues related to the pseudo r-square and maximum likelihood estimators, but the concept applies, and yet again goes back to the cultural conflicts between machine learning and traditional data modeling. Recall Breimer's quote from my post 'Culture Wars'

"Approaching problems by looking for a data model imposes an apriori straight jacket that restricts the ability of statisticians to deal with a wide range of statistical problems."

As I was cultured in the traditional data modeling paradigm, I've been more likely to think in terms of this straight jacket approach- critically thinking through the data model, the functional form, theoretical support for the varaibles to include and their functional relationship with y, etc. If your goal is to get coefficient estimates and make specific inferences about the relationships between specific variables in the model, then perhaps that is the approach to take. However, with hard deadlines, and large data sets, this becomes very time consuming and difficult. If your goal is actually prediction, as Breiman notes, you could easily paint yourself into a very restricted corner, and as Greene states, these need not be the same thing.

Ultimately it comes down to what your goals are and your client's preferences.

Scoring Data In SAS Enterprise Miner

The following diagram indicates the schema for scoring data in SAS Enterprise Miner. The SAS Code node is necessary for telling Enterprise Miner where the data to be scored is located, and where the model information is that will be used for scoring.  

/* CODE TO BE ENTERED IN THE SAS CODE NODE */
/* THE SCORE  DATA MUST BE IN A LIBRARY */
/* ACCESSIBLE TO ENTERPRISE MINER */

DATA YOUR_LIBRARY.YOUR_SCORE_DATA;
 SET &EM_IMPORT_SCORE;
 RUN;

  /* THE CODE IN THE SET STATEMENT  IS GENERIC CODE THAT SEEMS TO WORK
ANY WHERE IN THE PROJECT AS LONG AS IT FOLLOWS THE 'SCORE' NODE*/

Sample Assessment Score Rankings for SAS Forums

Interaction Models

Given a model of the form:

y= β₀ + β₁ X+ β₂Z  + β₃ XZ+ e

the relationship between X and Y is conditional on Z. The interaction term represents the effect of X on Y  conditional on the value of Z.

In ‘Understanding Interaction Models: Improving Empirical Analysis’ by Brambor, Clark, and Golder the following  schematic is presented:

 
As the schematic shows, β₂ represents the difference in intercepts between the two regression lines.

Notes:

Marginal Effect of X on Y: ∂Y/ ∂X = β₁  + β₃ Z

β₁ = effect of X on Y when Z =0

If XY is significant, that implies that the relationship between X and Y differs significantly between classes or values of Z.

It is possible that the effect of X on Y is significant for some values of Z even if the interaction term is not, hence you cannot base the inclusion of XZ in the model on the significance of the interaction term (Bramber et al, 2005).

In determining significance, the basic regression output typically does not provide sufficient information  and modifications are required (Bramber et al, 2005).

Kmenta (1971) provides the following comments regarding the significance of interactions and constitutive terms:

“When there are interaction terms in the equation, then any given explanatory variable may be represented not by one but several regressors. The hypothesis that this variable does not influence Y means that the coefficients of all regressors involving this variable are jointly zero”

As a result, the significance of X and the XZ term is given by the following F-test:

F = [ (R²₂ – R²₁) / (k₂ – k₁ )] / [(1-R²₂) / (N- k₂ -1)

K_n = # of variables in each model respectively (model including and excluding the interaction term and interaction variable)

R²_n = R-square for each respective model

N = total observations

The standard error of β₁  + β₃ Z = sqrt(V(β₁)  + Z² V(β₃ ) + 2 Z COV(β₁ ,β₃))

Constructing Odds Ratios from Logistic Models: e ^{β1  + β3 Z}

References:

Understanding Interaction Models: Improving Empirical Analyses. Thomas Bramber, William Roberts Clark, Matt Golder. Political Analysis (2006) 14:63-82

Elements of Econometrics. Jan Kmenta. Macmillan (1971)