Big Ag Meets Big Data Part II

Previously I discussed the role of social media in producing ‘big data’ and tools that may be used to get the most from this data in the ag industry. In this second installment I’m going to discuss other sources of ‘big data.’ 

I recall once  about 10 years ago attending a UK College of Agriculture field day in Princeton Ky, and someone made the comment that  went something like this: 

“these events are good because on the farm we don’t have time to set up experiments, collect data, and analyze to figure out best practices. We can’t stop and measure and record and report about everything we do.”

It’s certainly true that extension services will continue to conduct valuable research and it will probably remain a fact that producers aren’t going to necessarily have the time and resources to reduce their operation to a collection of well-crafted scientific experiments. However, every decision made on the farm is a trial of sorts, and with modern technology it is much easier to collect and log data about your operation, and some companies are now figuring out ways to take this farm level data and turn it into powerful analytical tools that can boost productivity and efficiency.    In a recent article ‘Building Big Data: Farming Big Data Goes To The Cows’ the following statement is made:

 

 "The major problem we keep on seeing — especially in bigger, modern farms — is that there's a lot of data being created and not being used, on how they're performing, what they're doing."

How is this data being generated? Lots if it is generated via your equipment including GPS:

“Next generation farm equipment like combines and tillers are going to be able to take soil samples as they move along, perform analysis on those samples, and feed the results of the analysis back to the manufacturer for crunching on a macro scale. This will result in a better understanding of what is happening in that entire area and make it possible to adjust things like the amount or types of fertilizer and chemicals that should be applied. If the farm equipment manufacturers figure out how to harness all this information, this kind of big-picture analysis could change the commodity trading markets forever." – from 4 Examples of Big Data Trends. Spetember 27,2012. VmwareBlogs.

 And how might we use this data?  Well some seed companies are already combining farm level data, public data, and their own proprietary data to develop some pretty powerful analytical tools. As discussed recently in an AgWeb technology article Steyer seeds offers a great example with its ACRES tool which is based on a complex form of decision tree:

“After they sign up, customers start by selecting their fields from Google Earth maps. Back-end programming then pulls up a wealth of information – everything from soil type to yield potential. As farmers enter in additional information about their farm, such as crop rotation, traits used, etc., the ACRES algorithm spits out recommendations, which users can accept or tweak as needed.” AgWeb - Unlock Your Farm Data

Another company, Climate Corporation is also taking advantage of massive amounts of data useful in agricultural applications:

"We took 60 years of crop yield data, and 14 terabytes of information on soil types, every two square miles for the United States, from the Department of Agriculture," says David Friedberg, chief executive of the Climate Corporation, …We match that with the weather information for one million points the government scans with Doppler radar — this huge national infrastructure for storm warnings — and make predictions for the effect on corn, soybeans and winter wheat." –New York Times

We’ve seen lots of efficiency, environmental, and productivity gains in agriculture related to GPS/GIS and biotechnology.  But with every trip across the field more and more data is being generated. Combining these technologies with ‘big data’ definitely will have its benefits, if not continue to revolutionize the industry.  

References and Further Reading:

Climate Corp. Updates Crop Insurance via High Tech. BloombergBusinessWeek. By Ashlee Vance on March 22, 2012.   http://www.businessweek.com/articles/2012-03-22/climate-corp-dot-updates-crop-insurance-via-high-tech

Big Data Goes to the Cows

From: http://siliconangle.com/blog/2012/09/07/building-big-data-farming-big-data-goes-to-the-cows/ 

Big Data in the Dirt (and the Cloud) October 11,2011. NYT. Quentin Hardy.

http://bits.blogs.nytimes.com/2011/10/11/big-data-in-the-dirt-and-the-cloud/

4 Examples of Big Data Trends. Spetember 27,2012. Vmware|Blogs.

http://blogs.vmware.com/vfabric/2012/09/4-examples-of-big-data-trends.html

Data analysis, biotech are key in agriculture's future sustainability

By Sarah Gonzalez

© Copyright Agri-Pulse Communications, Inc.

http://www.agri-pulse.com/ag-issues-biotech-future-22613.asp

Unlock Your Farm Data

February 15, 2013

By: Ben Potter, Farm Journal Technology Editor

http://www.agweb.com/article/unlock_your_farm_data/

Decision Trees and Gradient Boosting

Decision Trees

Decision tree algorithms search through the input space and find values of the input variables (split values) that maximize the differences in the target value between groups created by the split. The final model is characterized by the split values for each explanatory variable and creates a set of rules for classifying new cases.

Gradient Boosting

Boosting algorithms are ensemble methods that make predictions based on the average results of a series of weak learners. Gradient boosting involves fitting a series of trees, with each successive tree being fit to a resampled training set that is weighted according to the classification accuracy of the previously fit tree. The original training data is resampled several times and the combined series of trees form a single predictive model.  This differs from other ensemble methods using trees, such as random forests. Random forests are a modified type of bootstrap aggregation or bagging estimator (Freidman et al,2009). With random forests, we get a predictor that is an average of a series of trees grown on a bootstrap sample of the training data with only a random subset of the available inputs from the training data used to fit each tree (De Ville, 2006).  Gradient boosting can perform similarly to random forests and boosting may tend to dominate bagging methods in many applications. (Freidman et al,2009).

 References:

Friedman, Jerome H. (2001), Greedy function approximation: A gradient boosting machine. The Annals of Statistics, 29, 1189-1232. Available at http://stat.stanford.

Hasti, Tibshirani and Friedman. (2009)Elements of Statistical Learning: Data Mining,Inference, and Prediction. Second Edition. Springer-Verlag.

DeVille, Barry. (2006). Decision Trees for Business Intelligence and Data Mining Using SAS®  Enterprise Miner. SAS® Institute.

Big Ag Meets Big Data: Part 1

Social media has allowed farmers to organize and communicate about their industry.  The #agchat conversations on twitter are a good example. Not to mention Facebook (see Agriculture Proud for example) and YouTube ( like this look behind the scenes of a family farm). We've seen powerful examples of how social media can be used to mobilize voices and impact perceptions on a national level ( for example issues related to Yellow Tail wine and Pilot Travel Centers).

Social media also provides a rich data source for measuring sentiment or perceptions about the industry. Take for instance text mining. With Twitter, Facebook, email, online forums, open response surveys, customer and reader comments on web pages and news articles etc. there is a lot of information available to companies and organizations in the form of text. Without hiring experts to read through all of the thousands of pages worth of text available and making subjective claims about its meaning, text mining allows us to take otherwise unusable 'qualitative' data and convert it into quantitative measures that we can use for various types of reporting and modeling. Companies are finding that by mining text from web pages, comments, blogs, and social media, they can get measure consumer perceptions almost as well or better than they can through explicit surveys or other directly measurable outcomes in their databases. In my own personal experience, I've bench marked predictions made from traditional data base variables vs. text mining and found remarkable comparisons in performance.  The  validity of these  tools is not based necessarily on their ability to make new breakthrough discoveries, but on the contrary, how these algorithms give us almost exactly what we would expect, if we had time to manually process all of the information social media provides. (For a basic example of mining tweets related to 'factory farms' see: http://ageconomist.blogspot.com/2011/04/mining-tweets-abou-factory-farms.html ).

Besides the actual text we get from social media, the actual structure of social networks can also be very informative.  Social network analysis (SNA) allows us to answer questions such as who are key actors in a network? Who are the most influential members of a network? Who seems to be acting on the peripheral? Which connections in the network are most important?  Are there key players bridging connections or information between otherwise disconnected groups? Have policies or other forces changed the overall dynamics/interaction between people in the network (i.e. has the network structure changed in any meaningful way) and does that relate to some other performance outcome or goal? I’ve recently used this kind of information to help a company develop a predictive model to improve its viral marketing campaigns.

Of course, it doesn't take a rocket scientist to read tweets, Facebook posts, or blog comments to know when people are upset about a product. But there is also a wealth of knowledge to be gained from this type of information that is so voluminous, it would take an army of social media experts to glean and analyze. This is the essence of what has been termed in the industry as 'big data.' It requires new tools for capturing, storing, processing and analyzing this data, and a new type of analyst referred to as a data scientist.  These powerful analytics could be very beneficial to those in the ag industry or agvocacy groups. But this goes beyond social media, and I will discuss how big data is revolutionizing agriculture at the farm level in the second part of this two part series on big data.

*Note: I’m not using the term ‘big ag’ in the derogatory sense used by anti-agricultural activists, but in a complimentary sense referring to the complex network of modern family farms, biotechnology companies, food processors, other agribusinesses and retailers that cooperate to bring healthy and sustainable food to your table.

References:

Social Media Analytics. Matt Bogard, Applied Econometric and Analytical Consulting.
http://econometricsense.blogspot.com/2012/09/social-media-analytics.html

With Hadoop, Big Data Analytics Challenges Old-School Business Intelligence. Doug Henschen, Information Week
http://www.informationweek.com/software/business-intelligence/with-hadoop-big-data-analytics-challenge/240001922

Big Bets On Big Data. Eric Savitz, Forbes.  http://www.forbes.com/sites/ciocentral/2012/06/22/big-bets-on-big-data/

Creative Commons Image Attributions:

Handheld GPS

By Paul Downey from Berkhamsted, UK (Earthcache De Slufter  Uploaded by Partyzan_XXI) [CC-BY-2.0 (http://creativecommons.org/licenses/by/2.0)], via Wikimedia Commons

Satellite: NAVSTAR-2 (GPS-2) satellite Source: http://www.jpl.nasa.gov/images/grace/grace_083002_browse.jpg Status: PD-USGov-Military-Air Force {{PD-USGov-Military-Air Force}} Category:Satellites

Tractor: bdk [CC-BY-SA-3.0 (http://creativecommons.org/licenses/by-sa/3.0)], via Wikimedia Commons

 

Why does IFELSE logic work differently on what appear to be the same values?

 Embarrassingly I'm stumped on this...

I have a program in R for looking at grade distributions in my class. I found something weird recently with my 'ifelse' processing. I noticed that my program seemed to be over counting Cs and under counting Bs.

I'm not sure what's going on. It happened in the case where I was adding extra credit. It has something to do with the addition operation and variable assignment I guess. When I process untransformed data I get the correct # of Bs and Cs. But my goal in a program like this was flexibility - enter the data in excel and process. I'd prefer to add extra credit/curves/corrections etc. in R through processing vs. doing it all in excel.

 A very short modified  version (w/out reading in the data from csv) follows. Am I actually making a simple error in my math, or does R see these values differently than they may appear? I'm admittedly not a normal user of the 'ifelse' logic in R.  But even if it is wrong it should give me the same wrong answer when applied against what appear to be the same values! If I run the summary function  against all the vars I get matching results for grades$grade2 and grades$ec (the modified grade)

grades2 <- c(0.72,0.56,0.84,0.84,1.04,0.48,0.96,
0.8,0.68,0.92,0.72,0.6,0.92,0.72,0.88,0.88,0.76,
0.96,0.76,0.52,1,0.88,0.88,0.88,0.64)
 
grades1 <-c(0.64,0.48,0.76,0.76,0.96,0.4,0.88,
0.72,0.6,0.84,0.64,0.52,0.84,0.64,0.8,0.8,0.68,
0.88,0.68,0.44,0.92,0.8,0.8,0.8,0.56)
 
grades <- data.frame(cbind(grades1,grades2))
 
# format grades
 
grades$letter1 <- ifelse(grades$grades1 >= .90,"A", ifelse(grades$grades1 >= .80 & grades$grades1 < .90, "B",ifelse (grades$grades1 >= .70 & grades$grades1 < .80,"C",ifelse(grades$grades1 >= .60 & grades$grades1 < .70, "D","F"))))
 
# letter grade distribution
table(grades$letter1) 
 
grades$letter2 <- ifelse(grades$grades2 >= .90,"A", ifelse(grades$grades2 >= .80 & grades$grades2 < .90, "B",ifelse (grades$grades2 >= .70 & grades$grades2 < .80,"C",ifelse(grades$grades2 >= .60 & grades$grades2 < .70, "D","F"))))
 
# letter grade distribution
table(grades$letter2) 
 
# bonus: grade1 to = grade2
 
grades$ec <- grades1 + .08
 
# why is it that this misclassifies an .80 as a C in this case but not in the
# case of all of the previous grade1 and grade2 instances??
 
grades$letter.ec <- ifelse(grades$ec >= .90,"A", ifelse(grades$ec >= .80 & grades$ec < .90, "B",ifelse (grades$ec >= .70 & grades$ec < .80,"C",ifelse(grades$ec >= .60 & grades$ec < .70, "D","F"))))
 
# letter grade distribution
table(grades$letter.ec) 
 
summary(grades)
 
 
# if you print grades you can see that it missclassifies an .80 as a C
# for obs 8 for the calculated ec grade, but correctly classifies
# an .80 as  a B for cases of the grade1 and grade2 variables

Created by Pretty R at inside-R.org

Applied Analytics and Data Science Meet Precision Ag

Machine learning+ statistics+agronomics+economics = new line of business in applied data science. Monsanto is featured here, but the market is wide open. The most valuable data is being generated real time with every trip across the field. It's up to you to harness it and get the most value out of it. Privacy issues aside, it's not worth anything keeping it all to yourself. Don't just give it away, but don' t stuff your data under a mattress either. Welcome to the world of big data in big Ag.

"After they sign up, customers start by selecting their fields from Google Earth maps. Back-end programming then pulls up a wealth of information – everything from soil type to yield potential. As farmers enter in additional information about their farm, such as crop rotation, traits used, etc., the ACRES algorithm spits out recommendations, which users can accept or tweak as needed."

From: Farm Journal: Unlock Your Farm Data

http://www.agweb.com/article/unlock_your_farm_data/

Sounds like new opportunities for data scientists in the agriculture field.

That Modeling Feeling - a different take

I found the following post entitled SAS loves stats and really like this quote:

What advice would you give to students studying statistics today?

"Think about your favorite things in life – and what interests you the most.  You will likely find a role for statistics within those choices.  Once you start playing with the data related to your interests, you’ll learn quickly.  Data is everywhere.  Just be sure to look beyond the obvious."
 
And of course, a corollary to this is that if you love stats, then you might be able to find something interesting about almost anything. 

That modeling feeling « Statistical Modeling, Causal Inference, and Social Science

From: 
http://andrewgelman.com/2009/07/that_modeling_f/

This is exactly what it's like: 

"And the challenge is to get from point A to point B. So, you throw model after model at the problem, method after method, alternating between quick-and-dirty methods that get me nowhere, and elaborate models that give uninterpretable, nonsensical results. Until finally you get close. Actually, what happens is that you suddenly solve the problem! Unexpectedly, you're done! And boy is the result exciting. And you do some checking, fit to a different dataset maybe, or make some graphs showing raw data and model estimates together, or look carefully at some of the numbers, and you realize you have a problem. And you stare at your code for a long long time and finally bite the bullet, suck it up and do some active debugging, fake-data simulation, and all the rest. You code your quick graphs as diagnostic plots and build them into your procedure. And you go back and do some more modeling, and you get closer, and you never quite return to the triumphant feeling you had earlier—because you know that, at some point, the revolution will come again and with new data or new insights you'll have to start over on this problem..."

"But, not so deep inside you, that not-so-still and not-so-small voice reminds you of the compromises you've made, the data you've ignored, the things you just don't know if you believe. You want to do more, but that will require more computing, more modeling, more theory. Yes, more theory...."

And the post continues with more 'what its like' ...