Model Calibration and the Hosmer Lemeshaw Test

In a previous post I discussed the use of the ROC curve for model assessment. The ROC curve is a metric used to determine discrimination, how well a model discriminates between classes. Sometimes we are more interested in calibration, or the accurate stratification of individuals into higher or lower risk categories or risk strata.  In other words, sometimes we want to take predicted probabilities and divide our scored data set into groups, each having differing average predicted probabilities. A well calibrated model will make it possible to sort observations into strata or segments that exhibit an actual average outcome rate very close to the average predicted rate for the group. For example, the results below seem to indicate a well calibrated model for binary response Y ~ (0,1):

One might remark, these results look great, but are they statistically significant? The Hosmer-Lemeshaw test is a natural way to test for model calibration, and tests that very hypothesis. Without getting into great detail, the HL test is a chi-squared based test that divides your data set into deciles based on scores or predicted values and compares the observed response rates to the expected response rates (determined by your model's predicted probability of response). Significant differences between observed and predicted outcomes indicate lack of fit, so the HL test is a test for lack of fit i.e. significant values indicate lack of fit.

There are criticisms of the HL test, primarily related to its low power in small sample sizes, sensitivity to groupings, and its sensitivity to large sample sizes. Below are some examples of this last criticism:

JOURNAL OF PALLIATIVE MEDICINE
Volume 12, Number 2, 2009
Prediction of Pediatric Death in the Yearafter Hospitalization: A Population-Level  Retrospective Cohort Study Chris Feudtner, M.D., Ph.D., M.P.H.,1,5 Kari R. Hexem, M.P.H.,1 Mayadah Shabbout, M.S.,3James A. Feinstein, M.D.,1 Julie Sochalski, Ph.D., R.N.,4,5 and Jeffery H. Silber, M.D., Ph.D.2,5

‘The Hosmer-Lemeshow test detected a statistically significant degree of miscalibration in both models, due to the extremely large sample size of the models, as the differences between the observed and expected values within each group are relatively small.’ N = 600,000


Crit Care Med. 2007 Sep;35(9):2052-6.
Assessing the calibration of mortality benchmarks in critical care: The Hosmer-Lemeshow test revisited.

'Caution should be used in interpreting the calibration of predictive models developed using a smaller data set when applied to larger numbers of patients. A significant Hosmer-Lemeshow test does not necessarily mean that a predictive model is not useful or suspect. While decisions concerning a mortality model's suitability should include the Hosmer-Lemeshow test, additional information needs to be taken into consideration. This includes the overall number of patients, the observed and predicted probabilities within each decile, and adjunct measures of model calibration.'

Here we also have a cogent discussion of the HL test and its faults in the face of large sample sizes:

http://www.strath.ac.uk/aer/materials/5furtherquantitativeresearchdesignandanalysis/unit6/goodnessoffitmeasures/ 

"A disadvantage of this goodness of fit measure is that it is a significance test, with all the limitations this entails. Like other significant tests it only tells us whether the model fits or not, and does not tell us anything about the extent of the fit. Similarly, like other significance tests, it is strongly influenced by the sample size (sample size and effect size both determine significance), and in large samples, such as the PISA dataset we are using here, a very small difference will lead to significance. As the sample size gets large, the H-L statistic can find smaller and smaller differences between observed and model-predicted values to be significant. Small sample sizes are also problematic, however, as, being a Chi Square test we can’t have too many groups (more than 10%) with predicted frequencies of less than five."


What should we do in the face of large sample sizes and hypersensitive HL tests? Here is one suggestion: (link)

From: NATIONAL QUALITY FORUM National Voluntary Consensus Standards for Patient Outcomes Measure Summary.

“Because of the very large sample sizes studied here, a statistically significant
Hosmer-Lemeshow statistic is not considered informative with respect to calibration.”
Although the HL statistic is uninformative, model calibration could still be assessed graphically. This
could be done by comparing observed vs. predicted event rates within deciles of predicted risk."

The Area Under the ROC Curve

In a previous post I discussed predictive mode diagnostics in the context of the pseudo r-square statistic ( as well as the percentage of correct predictions) and also demonstrated the construction of an ROC curve.  However I did not expand on an interpretation of the ROC curve. Before discussing the ROC curve further,  let’s construct what is referred to as a confusion table that visualizes potential outcomes in a binary prediction scenario:

Some Definitions:

TP = true positive FP = false positive FN = false negative TN = true negative

% of Correct Predictions: For Y ~(0,1) the percentage of total correct predictions. (again see here for more details) or  (TP + TN) / (TP + FP + TN + FN)

Precision: Percentage of correctly predicted 1’s  or TP/(TP + FP))

Recall: Percentage of total observed or true 1’s correctly classified or TP/(TP+FN) also true positive rate

False Positive Rate:  FP /(FP + TN)

F1-Score: The harmonic mean of precision and recall or (2*Precision*Recall)/(Precision + Recall)

True positive rate: TP/(TP +FN) = Recall

Sensitivity: = recall

Specificity: = Percentage of total observed or true 0’s correctly classified or TN/(FP + TN)  or 1- false positive rate

All of the metrics mentioned above are based on classifying predictions based on a cutoff. If the predicted probability exceeds some threshold ‘c’ then we assign that observation a class value = 1. Otherwise the observation gets assigned a value = 0. These metrics are based on a single chosen cutoff. (one could examine multiple cutoffs and find the optimal value for c).

As explained in a previous post, the ROC curve is constructed by examining all possible cutoffs.  The ROC curve visualizes the tradeoffs between the true positive rate and false positive rate or sensitivity vs.  1-specificity. Particularly we are usually interested in the area under the ROC curve (AROC or c-statistic).  The ROC curve is a measure of a model’s discriminatory power.  The area under the ROC curve can be interpreted as the probability that a classifier will correctly rank a randomly chosen  training example with a positive outcome higher than a randomly chosen example with a negative outcome(Cook,2007).

This method is used increasingly in the machine learning community and is preferred over other measures of fit like precision or the F1-Score because it evaluates model performance across all considered cutoff values vs. an arbitrarily chosen cutoff (Bradley, 1997).It also gives a measure of classifier performance that gives low scores to random or one class only classifiers (Bradley,1997).

References: 
Bradley, Andrew P. Pattern Recognition, Volume 30, issue 7 (July, 1997), p. 1145-1159. Elsevier Science

Provost, F. J., Fawcett, T.,&  Kohavi, R. (1998). The Case against Accuracy Estimation for  Comparing Induction Algorithms. Proceedings of the Fifteenth  International Conference on Machine Learning (pp.445-453)(ICML '98), Jude W. Shavlik (Ed.). Morgan Kaufmann Publishers Inc.,San Francisco, CA, USA.

Nancy R. Cook, Use and Misuse of the Receiver Operating Characteristic Curve in Risk Prediction. Circulation. 2007; 115: 928-935

Tom Fawcett.. An introduction to ROC analysis.
Pattern Recognition Letters 27 (2006) 861–874

SAS Global Forum Papers

Using SAS® Enterprise BI and SAS® Enterprise MinerTM to Reduce Student Attrition

Matt Bogard, Western Kentucky University
Chris James
Tuesdi Helbig
Gina Huff

Abstract

The true supremacy of the SAS® Enterprise Business Intelligence Server is the ability to utilize the power of SAS® Analytics to deliver real-time information to end users, who usually do not understand statistics, but have the ability to make a difference if they have easy access to the analyzed data. This paper describes the process of using SAS® Enterprise Miner to develop a model to score university students based on their risk of attrition and deliver easy-to-understand results to university personnel using SAS® EBI.

Suggested Citation

Matt Bogard, Chris James, Tuesdi Helbig, and Gina Huff. "Using SAS® Enterprise BI and SAS® Enterprise MinerTM to Reduce Student Attrition" SAS Global Forum 2012 Proceedings.031-2012 (2012).

http://support.sas.com/resources/papers/proceedings12/031-2012.pdf 

A Data Driven Analytic Strategy for Increasing Yield and Retention at Western Kentucky University Using SAS Enterprise BI and SAS Enterprise Miner

Matt Bogard, Western Kentucky University

Article comments

SAS and all other SAS Institute Inc. product or service names are registered trademarks or trademarks of SAS Institute Inc. in the USA and other countries. ® indicates USA registration. Other brand and product names are trademarks of their respective companies.

Abstract

As many Universities face the constraints of declining enrollment demographics, pressure from state governments for increased student success, as well as declining revenues, the costs of utilizing anecdotal evidence and intuition based on ‘gut’ feelings to make time and resource allocation decisions become significant. However, grasping advanced statistical methods and analytics for data driven decision making can be overwhelming to some staff making buy in difficult. This paper describes how we are using SAS® Enterprise Miner to develop a model to score university students based on their probability of enrollment and retention early in the enrollment funnel so that staff and administrators can work to recruit students that not only have an average or better chance of enrolling but also succeeding once they enroll. Incorporating these results into SAS® EBI will allow us to deliver easy-to-understand results to university personnel.

 http://support.sas.com/resources/papers/proceedings13/044-2013.pdf

Topics of Interest

Because I just don't always have time to fully develop a post on everthing I come across, here are a few shorties:

Pauls Allison has done some great posts recently related to logistic regression and model assessment.

With regard to the pseudo R^2, see this post as well as the article associated with a new proposed alternative:

Tjur, T. (2009) “Coefficients of determination in logistic regression models—A new proposal: The coefficient of discrimination.” The American Statistician 63: 366-372.

(I've written about the pseudo R-square before here.) 

His most recent post discusses the Hosmer-Lemeshaw test. In the futrue I'd like to expand more on this, but he's critical of the test because it is sensitive to strata size. I am too, and I've also seen many criticisms related to its sensitivity to large sample sizes. I'll come back and expand more on that later, or do a separate post, but for now I'm just looking forward to his next article in which Paul is going to discuss some recent advancements and alternatives to the HL test.

Big Ag Meets Big Data Part II

Previously I discussed the role of social media in producing ‘big data’ and tools that may be used to get the most from this data in the ag industry. In this second installment I’m going to discuss other sources of ‘big data.’ 

I recall once  about 10 years ago attending a UK College of Agriculture field day in Princeton Ky, and someone made the comment that  went something like this: 

“these events are good because on the farm we don’t have time to set up experiments, collect data, and analyze to figure out best practices. We can’t stop and measure and record and report about everything we do.”

It’s certainly true that extension services will continue to conduct valuable research and it will probably remain a fact that producers aren’t going to necessarily have the time and resources to reduce their operation to a collection of well-crafted scientific experiments. However, every decision made on the farm is a trial of sorts, and with modern technology it is much easier to collect and log data about your operation, and some companies are now figuring out ways to take this farm level data and turn it into powerful analytical tools that can boost productivity and efficiency.    In a recent article ‘Building Big Data: Farming Big Data Goes To The Cows’ the following statement is made:

 

 "The major problem we keep on seeing — especially in bigger, modern farms — is that there's a lot of data being created and not being used, on how they're performing, what they're doing."

How is this data being generated? Lots if it is generated via your equipment including GPS:

“Next generation farm equipment like combines and tillers are going to be able to take soil samples as they move along, perform analysis on those samples, and feed the results of the analysis back to the manufacturer for crunching on a macro scale. This will result in a better understanding of what is happening in that entire area and make it possible to adjust things like the amount or types of fertilizer and chemicals that should be applied. If the farm equipment manufacturers figure out how to harness all this information, this kind of big-picture analysis could change the commodity trading markets forever." – from 4 Examples of Big Data Trends. Spetember 27,2012. VmwareBlogs.

 And how might we use this data?  Well some seed companies are already combining farm level data, public data, and their own proprietary data to develop some pretty powerful analytical tools. As discussed recently in an AgWeb technology article Steyer seeds offers a great example with its ACRES tool which is based on a complex form of decision tree:

“After they sign up, customers start by selecting their fields from Google Earth maps. Back-end programming then pulls up a wealth of information – everything from soil type to yield potential. As farmers enter in additional information about their farm, such as crop rotation, traits used, etc., the ACRES algorithm spits out recommendations, which users can accept or tweak as needed.” AgWeb - Unlock Your Farm Data

Another company, Climate Corporation is also taking advantage of massive amounts of data useful in agricultural applications:

"We took 60 years of crop yield data, and 14 terabytes of information on soil types, every two square miles for the United States, from the Department of Agriculture," says David Friedberg, chief executive of the Climate Corporation, …We match that with the weather information for one million points the government scans with Doppler radar — this huge national infrastructure for storm warnings — and make predictions for the effect on corn, soybeans and winter wheat." –New York Times

We’ve seen lots of efficiency, environmental, and productivity gains in agriculture related to GPS/GIS and biotechnology.  But with every trip across the field more and more data is being generated. Combining these technologies with ‘big data’ definitely will have its benefits, if not continue to revolutionize the industry.  

References and Further Reading:

Climate Corp. Updates Crop Insurance via High Tech. BloombergBusinessWeek. By Ashlee Vance on March 22, 2012.   http://www.businessweek.com/articles/2012-03-22/climate-corp-dot-updates-crop-insurance-via-high-tech

Big Data Goes to the Cows

From: http://siliconangle.com/blog/2012/09/07/building-big-data-farming-big-data-goes-to-the-cows/ 

Big Data in the Dirt (and the Cloud) October 11,2011. NYT. Quentin Hardy.

http://bits.blogs.nytimes.com/2011/10/11/big-data-in-the-dirt-and-the-cloud/

4 Examples of Big Data Trends. Spetember 27,2012. Vmware|Blogs.

http://blogs.vmware.com/vfabric/2012/09/4-examples-of-big-data-trends.html

Data analysis, biotech are key in agriculture's future sustainability

By Sarah Gonzalez

© Copyright Agri-Pulse Communications, Inc.

http://www.agri-pulse.com/ag-issues-biotech-future-22613.asp

Unlock Your Farm Data

February 15, 2013

By: Ben Potter, Farm Journal Technology Editor

http://www.agweb.com/article/unlock_your_farm_data/

Decision Trees and Gradient Boosting

Decision Trees

Decision tree algorithms search through the input space and find values of the input variables (split values) that maximize the differences in the target value between groups created by the split. The final model is characterized by the split values for each explanatory variable and creates a set of rules for classifying new cases.

Gradient Boosting

Boosting algorithms are ensemble methods that make predictions based on the average results of a series of weak learners. Gradient boosting involves fitting a series of trees, with each successive tree being fit to a resampled training set that is weighted according to the classification accuracy of the previously fit tree. The original training data is resampled several times and the combined series of trees form a single predictive model.  This differs from other ensemble methods using trees, such as random forests. Random forests are a modified type of bootstrap aggregation or bagging estimator (Freidman et al,2009). With random forests, we get a predictor that is an average of a series of trees grown on a bootstrap sample of the training data with only a random subset of the available inputs from the training data used to fit each tree (De Ville, 2006).  Gradient boosting can perform similarly to random forests and boosting may tend to dominate bagging methods in many applications. (Freidman et al,2009).

 References:

Friedman, Jerome H. (2001), Greedy function approximation: A gradient boosting machine. The Annals of Statistics, 29, 1189-1232. Available at http://stat.stanford.

Hasti, Tibshirani and Friedman. (2009)Elements of Statistical Learning: Data Mining,Inference, and Prediction. Second Edition. Springer-Verlag.

DeVille, Barry. (2006). Decision Trees for Business Intelligence and Data Mining Using SAS®  Enterprise Miner. SAS® Institute.