Erik Larson

Jul 9, 2010

The Wisdom of Crowds

I had an interesting discussion with one of my (many) liberal friends recently, and there’s a couple of points that came out of our prolonged exploration of every idea we could think of in the span of an evening. The first point worth sharing, I think, is as follows. The barnes and noble science section is packed with books explaining that we can’t predict the future in systems that aren’t governed by so-called normal distributions (though we think we can, a kind of persistent overconfidence in our epistemic abilities). The point here is that there’s all of this complexity in everyday life and social life (think economics), and we’re under the illusion that having a Ph.D. in economics and pointing to charts renders all of this moot. I’m talking about Taleb’s “Black Swan”, books like “The Drunkards Walk”, everything that Malcom Gladwell has every written, et cetera. So the point is, it turns out that huge parts of the world—and interestingly, ordinary parts of the world, like society and culture, politics, economics—are effectively black boxes with respect to prediction. We really just don’t know what tomorrow will bring. This has implications—huge implications—for the role of government or in general the role of experts in advising the rest of us on what courses of action should be undertaken. Much of this advice should properly be seen as speculation (there’s even research that suggests that experts are actually worse at predicting outcomes in complex “human” systems than non-experts).

So that’s point one. The other point is the “wisdom of crowds” notion, another concept that accounts for dozens of books in the BN science section (you know, where the smart-people-wanna-bes congregate). So this idea that many problems are solved by aggregating viewpoints, explored in books like, ah hum, The Wisdom of Crowds, Infotopia, Jeff Howe’s Crowdsourcing, et cetera, sugggests that having some expert decide things can be really stupid. In fact it turns out that groups—crowds—can often arrive at more optimal solutions to problems then even some one person who is educated and expert on solving problems of that type. It turns out, for instance, that allowing people to “bet” on some future outcome often produces the best prediction of that outcome. Asking an expert to figure out the outcome would result in a worse prediction. Just ordinary Joes (and of course experts too), if there’s enough of them, throwing down their money on which horse will win, or which companies to buy stock in, or which presidential candidate will win the election, often produces a better prediction on aggregate than the most expert horse person, or stock picker, or election pundit.

So this cutting edge research makes everyone feel all enlightened and egalitarian and up to date with the latest tidbits of “didn’t you know?” science. Only thing is, this is the best, greatest empirical argument for free markets ever. Let ordinary people figure out what to buy, where to shop, how the economy should go, from the ground up, so to speak. Science suggests that this libertarian technique often results in more optimal solutions then central planners sitting in government offices. This really amuses me, because the folks feeling all educated reading wisdom of the crowds literature— folks interested in social networking technology, reading Marx, sniffling about how republicans are idiots, eating tofu— yes these folks are in fact reading powerful arguments for individual liberty, limited government, the wisdom in crowds, not government planners trying to engineer the Good Society for everyone else. ( And they seem not to know it. Ha! ) The latter just isn’t optimal, if you believe the latest Gladwellesque arguments coming out of research on decision making.

Which brings me to my little wrap up. When I’m not sitting in cafes and I’m actually doing serious work, I’m reading Hayek, the Nobel economist who is widely credited as an intellectual precursor to libertarianism, particularly with regard to government involvement in the economy. Hayek, that Ph.D. egghead himself, who nonetheless argued (in my view persuasively) in the 1950s that because no one person can possibly know everything, the best strategy for a society is to vest more and more power in individuals. Hence individual liberty is a strategy that is most likely, over time, to result in more optimal solutions. Makes sense. The wisdom in crowds.

So the point is, again, that all of these insights emerging from the latest research in the social sciences are pointing back to non-government-controlled solutions to our most pressing problems. It’s pointing to a model where government’s most important job is to structure society in such a way that its citizens—all of us—can choose how best to live, what to buy, who to give money to, how to use our own money, and on and on. The aggregation of all of these individual voices makes things work better (right comrades ?). But of course people who are free to choose and to decide much of their lives on their own means that disparities in wealth and natural talents will result in disparities in society. And that bothers all of my liberal friends. “Can’t we just control people and liberate them too?” perhaps I heard (didn’t they tell us in school? you have to control people to try to make them equal, since we’re not naturally that way). But no I think unfortunately, on the aggregate, there’s a better and worse way to do things. And on the aggregate, their will always be winners and losers in the crowd.