Top News, Articles, and Interviews in Philosophy

Is X-Phi P-Hacked? (guest post by Mike Stuart, Edouard Machery and David Colaço)

Philosophy News image
Has experimental philosophy (“X-Phi”) exhibited signs of “p-hacking”? In this guest post*, Mike Stuart (Geneva), Edouard Machery (Pittsburgh), and David Colaço (Mississippi) report their findings. Is X-Phi P-Hacked? by Mike Stuart, with Edouard Machery and David Colaço Journals in psychology, neuroscience, and medicine pretty much only accept papers with significant p-values, usually setting the significance level at 0.05. Given that scientists are under immense pressure to publish often, and their papers will only be accepted if they report a p-value of 0.05 or lower, they may be tempted to make choices that help them reach this level. Without cooking the data, a significant p-value can be obtained in a number of ways, collectively known as p-hacking: You can perform statistical testing midway through a study to decide whether to collect more data (“optional stopping”); you can simply collect masses of data and then perform statistical tests on your data until something shows up (“data dredging”); you can drop outliers or rearrange treatment groups post hoc, etc. P-hacking is one of the main culprits for the replication crisis in psychology, neuroscience, and medicine. But what about experimental philosophy? Does it also suffer from p-hacking? In a paper just published in Analysis, David Colaço, Edouard Machery, and I examined a corpus of 365 experimental philosophy studies, which includes pretty much all the studies in x-phi from 1997 to 2016. We analyzed the p-values reported in these studies by p-curving them. A p-curve plots the frequency of significant p-values in a set of studies. Roughly, if scientists are p-hacking in order to reach significance, they probably aren’t going to the trouble of making their p-values much lower than 0.05. So, if most of the p values in a series of studies are hovering right below significance, we have reason to be suspicious. Once we have the p-curve, we can use its shape as an indicator of whether. . .

Continue reading . . .

News source: Daily Nous

blog comments powered by Disqus