Search Episodes
Listen, Share, & Support
Listen to the latest episode
Subscribe via iTunes
Subscribe via RSS
Become a fan
Follow on Twitter

Support Us:

Please consider making a donation to help make this podcast possible. Any contribution, great or small, helps tremendously!

Subscribe to E-Mail Updates

Related Readings
  • Answers for Aristotle: How Science and Philosophy Can Lead Us to A More Meaningful Life
    Answers for Aristotle: How Science and Philosophy Can Lead Us to A More Meaningful Life
    by Massimo Pigliucci
  • Nonsense on Stilts: How to Tell Science from Bunk
    Nonsense on Stilts: How to Tell Science from Bunk
    by Massimo Pigliucci
  • Denying Evolution: Creationism, Scientism, and the Nature of Science
    Denying Evolution: Creationism, Scientism, and the Nature of Science
    by Massimo Pigliucci

RS 209 - Christopher Chabris on "Collective intelligence & the ethics of A/B tests"

Release date: May 27th, 2018

Christopher Chabris

This episode features cognitive psychologist Christopher Chabris discussing his research on "collective intelligence" -- why do some teams perform better than others at a wide variety of tasks? Julia discusses potential objections to the findings and how gender-related publication bias should affect our interpretation of them. In the second half of the episode, Julia and Chris discuss why people get so upset at companies like Facebook and OKCupid for doing experiments on their users, and whether that's fair.

Chris' Website

Edited by Brent Silk

Music by Miracles of Modern Science


Full Transcripts 

Reader Comments (10)

What you said was right
May 27, 2018 | Unregistered Commenterwuxiaworld
I love this podcast. I need my twice monthly injection of rationality from RSPC to cope with all the nonsense I encounter on a daily basis.

Definitely socially aggressive people need to learn how to cooperate with others. A good leader will learn how to shut up and listen to others, and thereby maximize the benefit from the knowledge and skill of other people.

Julia makes such a good point about how a study that found groups benefit from more women would go to publication, but that a study that found women decrease the performance of groups would not go to publication. I would not want to be the fellow who publishes a study that shows women perform poorly in group performance or leadership roles, no matter how good the data !!

People probably dislike A/B Tests due to their inherent sense of Fairness. They misperceive A/B Tests as an unfair favoritism towards one group, and an unfair prejudice towards another. If two new promising therapies or processes develop, we need to compare the two with RCTs. Facebook should run A/B Tests. OKCupid almost necessarily has to run covert A/B Tests since their service involves wholly psychological results. Running a covert medical test involving treatment of a physical ailment without disclosing the possibility of a placebo would definitely create an ethical issue.

However, as Chabris mentions, if we have a new therapy or process A, or perhaps two new therapies A and B, and A and B clearly have superiority over the old standard therapy L, instead of requiring RCTs that include L versus A or B, we should only have RCTs with A versus B, and then compare those results to the Legacy Results from earlier RCTs, case reports, and other evidence of the efficacy of L. The FDA, however, usually strictly requires RCTs that include L, and frequently requires more than a decade of research before permitting widespread use of A or B. This is how the FDA kills a lot of people.

Implicit Bias Tests really don't work. Furthermore, all types of bias training usually fail. Really, Starbucks took the actions it took to save face and preserve its reputation. This has much more to due to marketing and brand protection, essential to customer retention and profits, than it has anything to do with a real concern for eliminating bias.

A General Disclosure of cover A/B experiments in every service agreement might make people feel better about the whole process. People would have to actually read the service agreement, or have someone point out this provision to them.
May 28, 2018 | Unregistered CommenterJameson
I just wanted to say that I thoroughly enjoyed this episode. The guest was great, and Julia did a beautiful job of the interview.
May 29, 2018 | Unregistered CommenterTristan
Apropos Dr. Chabris's admittedly limited recommendation of Malcom Gladwell's writing: I just happened on this trenchant take-down of Gladwell's methodology:

Gladwell's failure to consider an empirical test of his politically correct intervention in incarceration policy, see link above, is an excellent example of the failure of intellectuals to accept responsibility for the 'ideas' that they promulgate, and the failure to empirical test same, that Thomas Sowell and Nassim Taleb critique.

I find this piece of the Gladwell interview[1] telling:
<Begin quote ...>
Q. Your critics argue that your ideas are simplistic or overreliant on anecdotal evidence. Is that something you’ve taken to heart and tried to improve on?
A. Of course I rely on anecdotal evidence and simplify things; that’s what it means to be a popular writer.
<... end quote>

... my jaundiced ear hears, instead, 'you don't expect me to think-out complicated stuff and write it down for my audience of goodthinking humanities majors do you?

May 30, 2018 | Unregistered CommenterThurston
If a study finds that, say, women's startups fail more often, you know they'll chalk it up to sexism.
May 30, 2018 | Unregistered CommenterMax
I mean, if a study finds that women's startups fail more often, the study will chalk it up to sexism.
May 31, 2018 | Unregistered CommenterMax
<a href=””>cpl 2018 schedule</a>
June 3, 2018 | Unregistered
Nice opinion!
June 5, 2018 | Unregistered CommenterJSON formatter
Good one. A terrible thought occurred to me. If maybe sexism or alternatively the desire to avoid sexism, might skew experiments or the desire to do some experiments, can't this happen with almost any societal issue influencing many many experiments? Still, I think lack of incentives for reproduced experiments we've already had is a bigger problem than this.
June 15, 2018 | Unregistered CommenterChloe
great, thank you for sharing information
July 1, 2018 | Unregistered Commentershell shockers

PostPost a New Comment

Enter your information below to add a new comment.

My response is on my own website »
Author Email (optional):
Author URL (optional):
All HTML will be escaped. Hyperlinks will be created for URLs automatically.