Big Data and Hayek

A recent post by Matt Bogard drew my attention to a Forbes article entitled "Big Data Versus Hayek." Both Matt's post and the article are interesting, and I recommend reading both (they're short), but I want to pick a few nits with the Forbes article.

The authors mention a few examples of firms using big data methods to "set prices" and note that

What’s interesting about such centralized, algorithmic approach to price setting is how un-Hayekian it is.

This is an interesting point as far as it goes, but a couple of things should be noted. Hayek's point about centralized decision making was about markets, not firms. If complete decentralization were optimal, then no firm need exist. Of course, such an absurd conclusion can't be drawn from Hayek's work.

Firms are obviously necessary (and Matt makes some good points about this in relation to Coase), but Hayek's point is about the markets in which firms operate. Decentralized markets generate prices that reflect the availability of resources needed for production and the tastes and preferences of consumers; these prices allow firms to provide what consumers want. The USSR was (largely) without the coordinating effect of prices, a key cause of its demise.

The second problem is the idea that these firms are "setting prices." Surely a price tag or the rate offered on the Uber app will in some sense be "set" by the firm. However, the firm has very little control over the price they receive because ultimately the consumer determines the equilibrium price to which actual prices continually move. The influence of regulations, competitors, and (perhaps most powerfully) the preferences of consumers will decide what price the firm receives. The firm can "set" any price it likes, but decentralized free markets ensure that these prices come to closely approximate the value consumers place on the product.

The author concludes

So is big data displacing Hayek? In some places, I would say yes, for sure. But we should not get ahead of ourselves in declaring the death of decentralized knowledge and decision-making. As a prediction, pricing, and choice-making device, algorithms and big data still have a long way to go in many contexts. And rather than being left in the dust, markets themselves are evolving too.

The problem here is that a key part of Hayek's contribution to the economics of information is that important aspects of the market economy can't be measured and thus can't be incorporated in big data methods. In his Nobel Prize Lecture entitled "The Pretense of Knowledge" Hayek said:

Unlike the position that exists in the physical sciences, in economics and other disciplines that deal with essentially complex phenomena, the aspects of the events to be accounted for about which we can get quantitative data are necessarily limited and may not include the important ones. While in the physical sciences it is generally assumed, probably with good reason, that any important factor which determines the observed events will itself be directly observable and measurable, in the study of such complex phenomena as the market, which depend on the actions of many individuals, all the circumstances which will determine the outcome of a process, for reasons which I shall explain later, will hardly ever be fully known or measurable. And while in the physical sciences the investigator will be able to measure what, on the basis of a prima facie theory, he thinks important, in the social sciences often that is treated as important which happens to be accessible to measurement. This is sometimes carried to the point where it is demanded that our theories must be formulated in such terms that they refer only to measurable magnitudes.

Hayek is talking here about economists and their ability to examine the real world economy, but I think there's an important point here for firms as well. Big data methods will most certainly help managers and entrepreneurs make predictions about the best way to position their firms. However, Hayek's "circumstances of time and place" mentioned in the Forbes piece are likely to be important and cannot be incorporated into big data models because they can't be measured. In this sense, then, the juxtaposition of Hayek's contributions to the economics of information to the development and implementation of big data methods misunderstands, to an extent, Hayek's point.