Monday, March 4, 2013

Big Ag Meets Big Data: Part 1

Social media has allowed farmers to organize and communicate about their industry.  The #agchat conversations on twitter are a good example. Not to mention Facebook (see Agriculture Proud for example) and YouTube ( like this look behind the scenes of a family farm). We've seen powerful examples of how social media can be used to mobilize voices and impact perceptions on a national level ( for example issues related to Yellow Tail wine and Pilot Travel Centers).

Social media also provides a rich data source for measuring sentiment or perceptions about the industry. Take for instance text mining. With Twitter, Facebook, email, online forums, open response surveys, customer and reader comments on web pages and news articles etc. there is a lot of information available to companies and organizations in the form of text. Without hiring experts to read through all of the thousands of pages worth of text available and making subjective claims about its meaning, text mining allows us to take otherwise unusable 'qualitative' data and convert it into quantitative measures that we can use for various types of reporting and modeling. Companies are finding that by mining text from web pages, comments, blogs, and social media, they can get measure consumer perceptions almost as well or better than they can through explicit surveys or other directly measurable outcomes in their databases. In my own personal experience, I've bench marked predictions made from traditional data base variables vs. text mining and found remarkable comparisons in performance.  The  validity of these  tools is not based necessarily on their ability to make new breakthrough discoveries, but on the contrary, how these algorithms give us almost exactly what we would expect, if we had time to manually process all of the information social media provides. (For a basic example of mining tweets related to 'factory farms' see: ).

Besides the actual text we get from social media, the actual structure of social networks can also be very informative.  Social network analysis (SNA) allows us to answer questions such as who are key actors in a network? Who are the most influential members of a network? Who seems to be acting on the peripheral? Which connections in the network are most important?  Are there key players bridging connections or information between otherwise disconnected groups? Have policies or other forces changed the overall dynamics/interaction between people in the network (i.e. has the network structure changed in any meaningful way) and does that relate to some other performance outcome or goal? I’ve recently used this kind of information to help a company develop a predictive model to improve its viral marketing campaigns.

Of course, it doesn't take a rocket scientist to read tweets, Facebook posts, or blog comments to know when people are upset about a product. But there is also a wealth of knowledge to be gained from this type of information that is so voluminous, it would take an army of social media experts to glean and analyze. This is the essence of what has been termed in the industry as 'big data.' It requires new tools for capturing, storing, processing and analyzing this data, and a new type of analyst referred to as a data scientist.  These powerful analytics could be very beneficial to those in the ag industry or agvocacy groups. But this goes beyond social media, and I will discuss how big data is revolutionizing agriculture at the farm level in the second part of this two part series on big data.

*Note: I’m not using the term ‘big ag’ in the derogatory sense used by anti-agricultural activists, but in a complimentary sense referring to the complex network of modern family farms, biotechnology companies, food processors, other agribusinesses and retailers that cooperate to bring healthy and sustainable food to your table.


Social Media Analytics. Matt Bogard, Applied Econometric and Analytical Consulting.

With Hadoop, Big Data Analytics Challenges Old-School Business Intelligence. Doug Henschen, Information Week
Big Bets On Big Data. Eric Savitz, Forbes.

Creative Commons Image Attributions:
Handheld GPS
By Paul Downey from Berkhamsted, UK (Earthcache De Slufter  Uploaded by Partyzan_XXI) [CC-BY-2.0 (], via Wikimedia Commons
Satellite: NAVSTAR-2 (GPS-2) satellite Source: Status: PD-USGov-Military-Air Force {{PD-USGov-Military-Air Force}} Category:Satellites
Tractor: bdk [CC-BY-SA-3.0 (], via Wikimedia Commons

No comments:

Post a Comment