RS76 - Crowdsourcing and the Wisdom of Crowds

Release date: December 16, 2012

What do Linux, Netflix, and the Oxford English Dictionary have in common? They've all benefited from the power of crowdsourcing, in which a task is outsourced to a group of hundreds or thousands of disparate people. In this episode of Rationally Speaking, Julia and Massimo discuss the phenomena of crowdsourcing, and ask: What makes it work? Is it ever unethical? And what are the limits to the wisdom of crowds?

Julia's pick: "The Signal and the Noise: Why So Many Predictions Fail-but Some Don't"

Massimo's pick: The Phi2Phi App.

References:
"Longitude" 

Comments

[Max]

There's a study on wisdom of the crowds going on now, called Forecasting ACE. They found that information sharing improves forecasts over independent forecasts. Perhaps because, as you said, it gets people to consider counterarguments.

From the study:

One experiment investigated whether information sharing among the contributors would be helpful or harmful to performance. On the one hand, there is evidence that forecasts resulting from discussions tend to be more similar and therefore reflect less varied input. On the other hand, discussions can also cause deeper thought. Discussions aren't practical on the ForecastingAce site, but we instituted simple sharing of rationales. The question was, would that help or hurt performance? For a period of time, half the contributors were invited to read other people's rationales and half were not. The results were clear: the ability to read other people's rationales (sharing information) improved performance. Consequently, information sharing became the default on the website on June 13, 2012.

Specifically, the results of the experiment were:
Contributors in the information-sharing group revised their forecasts more frequently, meaning they reflected more up-to-date information.
The Brier Score for the information-sharing group improved relative to the non-information-sharing group.
Those in the information-sharing group tended to begin forecasting earlier in the life of a problem, suggesting they were more engaged. On the negative side, unless they made later revisions in their confidence estimates (which many did), their forecasts tended to be less accurate, as there is strong evidence that forecasts are more accurate closer to the end date.

[Gil]

An interesting podcast.

I want to mention the most obvious reason why people contribute to Wikipedia and such projects that you didn't mention in the podcast. People are simply passionate about a topic and decide to share their knowledge. They need not to be professional experts but they can be very knowledgeable about it. Also, there are now several initiatives in the academia to promote high quality writings in Wikipedia. In psychology for example, there are professors who as part of the class, assign students to certain subjects. Instead of writing a boring research paper that no one is going to read, students can actually contribute valuable information to the public.

[Michael Zacharie]

Can you guys do an episode on the beauocracy of academia?

[Michael Zacharie]

Can you guys do an episode on the beauocracy of academia?

[Ihm]

Debian, Ubuntu, etc. are distributions of Linux. :)

[ABQOkami]

I don't believe there is much need to say what a great contribution Massimo and Julia make to the conversation about rationality. So many have done that before. However (in the great parenthetical tradition of Massimo's wrting: you knew there was a "however" coming), I have become hung up on a recent habit of Julia's and it's making listening to the podcasts harder and harder for someone with a light case of OCD. I refer to her constant over-use and often misuse of the work "like". On this podcast I counted well over 50 "like"s (I was riding my MTB and lost count while dodging rocks). In my humble opinion it greatly diminishes the accomplishments of what is obviously a wonderful, well-read, and thoughtful mind. Please, please try to limit this one, minor bad habit, and you will have as close to a perfect podcast as anyone could want. Of course, Julia can take heart that she is in great company: just listen to Ira Glass speak extemporaneously or any broadcast by Teri Gross. All my love, a faithful listener.

[Yatin]

It's Really one of the Awesome article i have read it. Thanks for the article. Also i have made a Blog on Good Friday 2015 & Good Friday Images 2015. Thanks and Regards, Ayush Sharma

[Jameson]

Hosting a contest to solve a difficult problem or present an innovative design makes good sense since the judges, and perhaps the contestants of the contest, can learn from the various submissions.