Experimentation and Causal Inference Meet the Knowledge Problem

Why should firms leverage experimentation and causal inference? With recent advancements in computing power and machine learning, why can't they simply base all of their decisions on predictions or historical patterns discovered in the data using AI?  Perhaps statisticians and econometricians and others have a simple answer. The kinds of learnings that will help us understand the connections between decisions and the value we create require understanding causality. This requires something that may not be in the data to begin with. Experimentation and causal inference may be the best (if not the only) way of answering these questions. In this series of posts I want to focus on a number of fundamental reasons that experimentation and causal inference are necessary in business settings from the perspective of both mainline and behavioral economics:

Part 1: The Knowledge Problem
Part 2:  Behavioral Biases
Part 3:  Strategy and Tactics

In this post I want to discuss the value of experimentation and causal inference from a basic economic perspective. The fundamental problem of economics, society, and business is the knowledge problem. In his famous 1945 American Economic Review article The Use of Knowledge in Society, Hayek argues:

"the economic problem of society is not merely a problem of how to allocate 'given resources'....it is a problem of the utilization of knowledge which is not given to anyone in its totality."

A really good parable explaining the knowledge problem is the essay I, Pencil by Leonard E. Read. The fact that no one person possesses the necessary information to make something that seems so simple as a basic number 2 pencil captures the essence of the knowledge problem.

If you remember your principles of economics, you know that the knowledge problem is solved by prices which reflect tradeoffs based on the disaggregated incomplete and imperfect knowledge and preferences of millions (billions) of individuals. Prices serve both the function of providing information and the incentives to act on that information. It is through this information creation and coordinating process that prices help solve the knowledge problem. Prices solve the problem of calculation that Hayek alluded to in his essay, and they are what coordinate all of the activities discussed in I, Pencil. 

In Living Economics: Yesterday, Today, and Tommorow by Peter J. Boettke, discusses the knowledge problem in the context of firms and the work of economist Murray Rothbard:

"firms cannot vertically integrate without facing a calculation problem....vertical integration eliminates the external market for producer goods."

 Coase, also recognized that as firms integrate to eliminate transactions costs they also eliminate the markets which generate the prices that solve the knowledge problem! This tradeoff has to be managed well or firms go out of business. In a way firms could be viewed as little islands with socially planned economies in a sea of market competition. As Luke Froeb masterfully illustrates in his text Managerial Economics: A Problem Solving Approach (3rd Ed), decisions within firms in effect create regulations, taxes, and subsidies that destroy wealth creating transactions. Managers should make decisions that consummate the most wealth creating transactions (or do their best not to destroy, discourage, or prohibit wealth creating transactions).

So how do we solve the knowledge problems in firms without the information creating and coordinating role of prices? Whenever mistakes are made, Luke Froeb provides this problem solving algorithm that asks:

1) Who is making the bad decision?
2) Do they have enough information to make a good decision?
3) Do they have the incentive to make a good decision?

In essence, in absence of prices, we must try to answer the same questions that market processes often resolve. And we could leverage experimentation and causal inference to address each of the questions above:

How do we know a decision was good or bad to begin with? 

How do we get the information to make a good decision? 

What incentives or nudges work best to motivate good decision making? 

What does failure to solve the knowledge problem in firms look like in practical terms? Failure to consummate wealth creating transactions implies money left on the table - but experimentation and causal inference can help us figure out how to reclaim some of these losses. List and Gneezy address this in The Why Axis:

"We think that businesses that don't experiment and fail to show, through hard data, that their ideas can actually work before the company takes action - are wasting their money....every day they set suboptimal prices, place adds that do not work, or use ineffective incentive schemes for their work force, they effectively leave millions of dollars on the table."

Going back to I, Pencil and Hayek's essay, the knowledge problem is solved through the spontaneous coordination of multitudes of individual plans via markets. Through a trial and error process where feedback is given through prices, the plans that do the best job coordinating peoples choices are adopted. Within firms there are often only a few plans compared to the market and these are in the form of various strategies and tactics. But as discussed in Jim Manzi's book Uncontrolled, firms can mimic this trial and error feedback process through iterative experimentation.

While experimentation and causal inference cannot perfectly emulate the same kind of evolutionary feedback mechanisms prices deliver in market competition, an iterative test and learn culture within a business may provide the best strategy for dealing with the knowledge problem. And that is one of many ways that experimentation and causal inference can create value.

Read Part 2: Behavioral Biases

See also:

Statistics is a Way of Thinking, Not a Box of Tools