Why does IFELSE logic work differently on what appear to be the same values?

 Embarrassingly I'm stumped on this...

I have a program in R for looking at grade distributions in my class. I found something weird recently with my 'ifelse' processing. I noticed that my program seemed to be over counting Cs and under counting Bs.

I'm not sure what's going on. It happened in the case where I was adding extra credit. It has something to do with the addition operation and variable assignment I guess. When I process untransformed data I get the correct # of Bs and Cs. But my goal in a program like this was flexibility - enter the data in excel and process. I'd prefer to add extra credit/curves/corrections etc. in R through processing vs. doing it all in excel.

 A very short modified  version (w/out reading in the data from csv) follows. Am I actually making a simple error in my math, or does R see these values differently than they may appear? I'm admittedly not a normal user of the 'ifelse' logic in R.  But even if it is wrong it should give me the same wrong answer when applied against what appear to be the same values! If I run the summary function  against all the vars I get matching results for grades$grade2 and grades$ec (the modified grade)

grades2 <- c(0.72,0.56,0.84,0.84,1.04,0.48,0.96,
0.8,0.68,0.92,0.72,0.6,0.92,0.72,0.88,0.88,0.76,
0.96,0.76,0.52,1,0.88,0.88,0.88,0.64)
 
grades1 <-c(0.64,0.48,0.76,0.76,0.96,0.4,0.88,
0.72,0.6,0.84,0.64,0.52,0.84,0.64,0.8,0.8,0.68,
0.88,0.68,0.44,0.92,0.8,0.8,0.8,0.56)
 
grades <- data.frame(cbind(grades1,grades2))
 
# format grades
 
grades$letter1 <- ifelse(grades$grades1 >= .90,"A", ifelse(grades$grades1 >= .80 & grades$grades1 < .90, "B",ifelse (grades$grades1 >= .70 & grades$grades1 < .80,"C",ifelse(grades$grades1 >= .60 & grades$grades1 < .70, "D","F"))))
 
# letter grade distribution
table(grades$letter1) 
 
grades$letter2 <- ifelse(grades$grades2 >= .90,"A", ifelse(grades$grades2 >= .80 & grades$grades2 < .90, "B",ifelse (grades$grades2 >= .70 & grades$grades2 < .80,"C",ifelse(grades$grades2 >= .60 & grades$grades2 < .70, "D","F"))))
 
# letter grade distribution
table(grades$letter2) 
 
# bonus: grade1 to = grade2
 
grades$ec <- grades1 + .08
 
# why is it that this misclassifies an .80 as a C in this case but not in the
# case of all of the previous grade1 and grade2 instances??
 
grades$letter.ec <- ifelse(grades$ec >= .90,"A", ifelse(grades$ec >= .80 & grades$ec < .90, "B",ifelse (grades$ec >= .70 & grades$ec < .80,"C",ifelse(grades$ec >= .60 & grades$ec < .70, "D","F"))))
 
# letter grade distribution
table(grades$letter.ec) 
 
summary(grades)
 
 
# if you print grades you can see that it missclassifies an .80 as a C
# for obs 8 for the calculated ec grade, but correctly classifies
# an .80 as  a B for cases of the grade1 and grade2 variables

Created by Pretty R at inside-R.org

Applied Analytics and Data Science Meet Precision Ag

Machine learning+ statistics+agronomics+economics = new line of business in applied data science. Monsanto is featured here, but the market is wide open. The most valuable data is being generated real time with every trip across the field. It's up to you to harness it and get the most value out of it. Privacy issues aside, it's not worth anything keeping it all to yourself. Don't just give it away, but don' t stuff your data under a mattress either. Welcome to the world of big data in big Ag.

"After they sign up, customers start by selecting their fields from Google Earth maps. Back-end programming then pulls up a wealth of information – everything from soil type to yield potential. As farmers enter in additional information about their farm, such as crop rotation, traits used, etc., the ACRES algorithm spits out recommendations, which users can accept or tweak as needed."

From: Farm Journal: Unlock Your Farm Data

http://www.agweb.com/article/unlock_your_farm_data/

Sounds like new opportunities for data scientists in the agriculture field.

That Modeling Feeling - a different take

I found the following post entitled SAS loves stats and really like this quote:

What advice would you give to students studying statistics today?

"Think about your favorite things in life – and what interests you the most.  You will likely find a role for statistics within those choices.  Once you start playing with the data related to your interests, you’ll learn quickly.  Data is everywhere.  Just be sure to look beyond the obvious."
 
And of course, a corollary to this is that if you love stats, then you might be able to find something interesting about almost anything. 

That modeling feeling « Statistical Modeling, Causal Inference, and Social Science

From: 
http://andrewgelman.com/2009/07/that_modeling_f/

This is exactly what it's like: 

"And the challenge is to get from point A to point B. So, you throw model after model at the problem, method after method, alternating between quick-and-dirty methods that get me nowhere, and elaborate models that give uninterpretable, nonsensical results. Until finally you get close. Actually, what happens is that you suddenly solve the problem! Unexpectedly, you're done! And boy is the result exciting. And you do some checking, fit to a different dataset maybe, or make some graphs showing raw data and model estimates together, or look carefully at some of the numbers, and you realize you have a problem. And you stare at your code for a long long time and finally bite the bullet, suck it up and do some active debugging, fake-data simulation, and all the rest. You code your quick graphs as diagnostic plots and build them into your procedure. And you go back and do some more modeling, and you get closer, and you never quite return to the triumphant feeling you had earlier—because you know that, at some point, the revolution will come again and with new data or new insights you'll have to start over on this problem..."

"But, not so deep inside you, that not-so-still and not-so-small voice reminds you of the compromises you've made, the data you've ignored, the things you just don't know if you believe. You want to do more, but that will require more computing, more modeling, more theory. Yes, more theory...."

And the post continues with more 'what its like' ...

Paul Allison On Multicollinearity

From:  http://www.statisticalhorizons.com/multicollinearity

I happened to stumble upon this great article by Paul Allison who I had the priviledge of seeing at last year's SAS Global Forum. Actually, everything I've read by him from logistic regression to survival analysis and data imputation is very thorough and cogent. Here are a few of his points that hit home with me: 

When is it OK to ignore multicollinearity? Well when....

# 1. The variables with high VIFs are control variables, and the variables of interest do not have high VIFs.

"Here's an example from some of my own work: the sample consists of U.S. colleges, the dependent variable is graduation rate, and the variable of interest is an indicator (dummy) for public vs. private. Two control variables are average SAT scores and average ACT scores for entering freshmen. These two variables have a correlation above .9, which corresponds to VIFs of at least 5.26 for each of them. But the VIF for the public/private indicator is only 1.04. So there's no problem to be concerned about, and no need to delete one or the other of the two controls."

# 3. The variables with high VIFs are indicator (dummy) variables that represent a categorical variable with three or more categories.

You say Stata I say SAS: Software Signaling and Social Identity Theory

In my previous post I quoted the following: 

"When you don't have to code your own estimators, you probably won't understand what you're doing. I'm not saying that you definitely won't, but push-button analyses make it easy to compute numbers that you are not equipped to interpret." 

Then I said: 

I Agree. Statistics is a language best communicated and understood via code vs. a point and click GUI.

But I revised the post and restated:

I agree that statistics is a language best communicated and understood via code vs. a point and click GUI. 

I've been thinking a lot about this, and considering some insight from Rick Wicklin in the comments to that post. What I should actually say is that  *personally* I don't feel like I understand an estimator as well until I've actually coded it (as an algorithm not just submitting a command to a software package) or at least made some attempt to implement it in some simplified way, or I get some idea of how it could be coded if my coding skills were up to the challenge. So, by that measure I understand some estimators better than others and trying to better understand estimators in this way is really the purpose of this blog. 

But I've thought a little more about the first quote. *IF* I understand correctly how SAS/R/STATA/SPSS etc. works, regardless if you are pointing and clicking via a GUI interface or submitting canned routines at the command line, *BOTH* are wrappers for the heavy lifting and actual statistical programming abstracted behind the scenes by developers. Both command line and GUI environments  make it pretty easy to get results you may or may not correctly interpret, or simply to apply the wrong test in the wrong situation.

But, you can also really get into trouble actually coding your estimator (maybe using SAS IML or R or Octave). If you don't know what you are doing and you click through a regression in SPSS or submit a PROC in SAS, at least the estimates will be correct. If you make a syntax error the code likely just won't run and the log may even help you out.  The application of the correct statistical test or interpretation is up to you. But you can make a mistake coding your own estimator and not even realize it. Efficiency, repeatability, resource requirements etc. all factor in too.

As far as 'signaling' goes, now that I've thought more about it, a screen full of code certainly may give the appearance of a more sophisticated analysis and may even give one a false sense of confidence in the results i.e. the signal for sophistication or quality may be mixed at best. 

In a recent blog post, Andrew Gelman states (in the context of this same software signal discussion on his blog): 

  To me, a statistics package is not just its code, it’s also its community, it’s what people do with it.

I can relate to the community aspect.  On campus, within academia we have our different communities of users (Stata, SPSS, Excel, SAS,  Mathematica ,and some R) and I think Rick's comments on the previous post are informative about academic and business communities.  I think lots of times, social identity theory begins to play out;  we get really comfortable within our community and  tend to pigeonhole and short change other tools, and even worse project those perceptions onto the users of those tools.  I've been guilty of this and Rick helped me realize it.

 

**************************

Note: There can definitely be benefits to coding your own estimators. If you are a coder  and use SAS, I highly recommend  Rick Wicklin's The Do Loop where he has produced a number of excellent posts explaining the nuts and bolts of coding your own estimators and gets behind the scenes of a vast array of concepts (like the power method for computing only the largest eigenvalue and this tip on NOT using macros to code a simulation.  See also 12 Tips for SAS Statistical Programmers from 2012.

Decomposition: The Statistics Software Signal

From:

Decomposition: The Statistics Software Signal

http://seanjtaylor.com/post/39573264781/the-statistics-software-signal

"When you don't have to code your own estimators, you probably won't understand what you're doing. I'm not saying that you definitely won't, but push-button analyses make it easy to compute numbers that you are not equipped to interpret."

I agree that statistics is a language best communicated and understood via code vs. a point and click GUI.

However, particularly interesting is his view of how the use of a given software package may relate to the quality of research:

"SPSS: You love using your mouse and discovering options using menus. You are nervous about writing code and probably manage your data in Microsoft Excel." (see the linked article for similar remarks)


 To be fair, STATA, SPSS, SAS and R have coding environments, and as a user of both SAS and R products I don't see why using PROC REG in SAS is any less sophisticated than the 'lm' function in R. Nor do I see any difference in coding an estimator or algorithm in R vs. SAS IML.

In fact, there has been a long running discussion for over a year now on SAS vs. R on LinkedIn and in my opinion it all it has established is that R certainly provides a powerful software solution for many researchers and businesses. 

It would be interesting to quantify and test Taylor's theory.

UPDATE: see You say Stata I Say SAS: software signaling and social identity theory.